Installation

Lenses is available in a number of different deployment options from linux archive to Kubernetes. Installation is straightforward but the approach differs based on your Apache Kafka.

  1. For a quick local demo experience chose the Lenses box. This contains a broker and everything you need to test out Lenses features.
  2. To connect Lenses to your existing self-managed Kafka cluster, use the Lenses docker or Linux archive. On start up Lenses will guide you through connecting to your brokers and zookeeper and setting up JMX monitoring.
  3. For Kubernetes, you can use our Helm chart.
  4. For managed Kafka’s follow the appropriate deployment guides.

For more advanced installs:

  1. Check the prerequisites for your infrastructure.
  2. Understand how to configure Lenses for your Kafka and other services.
  3. Undestand the recommendations for production use.

Lenses Architecture 

Lenses is architected to enable different teams and engineers to work with data in real-time and create applications and flows. Security and governance are built-in components to enable developers and admins to create secure workspaces and automations.

The diagram gives a high-level overview of the logical components. At the core of Lenses, we have:

  • SQL Engine to query data and create streaming apps leveraging Kafka Streams
  • App Engine to manage seamless deployments of SQL apps (deployed to Kubernetes)
  • Metadata Engine to create a real-time Data Catalog for cross-system datasets and apps

Lenses is a JVM application, written in Scala and exposes secure restful APIs and websockets in addition to providing a user interface.

The UI, developed in React, is served by Akka built-in HTTP webserver.

Lenses architecture

Lenses requires the following for a setup:

  1. A configuration file - defining low level configuration settings, for example Akka settings
  2. A security file - defining the authentication configuration for Lenses This setup one or more authentication modules for Lenses, e.g. SSO, Basic, LDAP
  3. An embedded / backing database or an external Postgres database This holds the Lenses state, application data, and audit logs

Prerequisites 

Kafka versions 

Any version of Apache Kafka (2.0 or newer) on-premise and on-cloud.

JMX connectivity 

For performance and health monitoring, optionally enable JMX metrics.

Hardware & OS 

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

Copy
ulimit -S -n     # soft limit
ulimit -H -n     # hard limit

Increase as a super-user the soft limit to 4096 with:

Copy
ulimit -S -n 4096

Use 6GB RAM/4 CPUs and 500MB disk space.

Browser 

All recent versions of major browsers are fully supported.

APIs and Websockets 

Every action in Lenses is backed by an API or websocket, documented at https://api.lenses.io. A Golang client is available and CLI (command line interface).

Lenses state store 

Lenses can use an embedded H2 database or a Postgres database. Postgres is not supplied by Lenses.

JMX 

Connectivity to JMX is optional (not required) but recommended for additional/enhanced monitoring of the Kafka Brokers and Connect Workers. Secure JMX connections are also supported, as well as JOLOKIA and OpenMetrics (MSK).

TLS termination 

By default, Lenses does not provide TLS termination but can be enabled via a configuration option. TLS termination is recommended for enhanced security and a prerequisite for integrating with SSO (Single Sign On) via SAML2.0.

TLS termination can be configured directly within Lenses or alternatively by using a TLS proxy or load balancer. Refer to the TLS documentation for additional information.