Prerequisites

Kafka Versions

Lenses 2.3 supports Kafka from version 1.0 up to the latest Kafka version, which at the time of release is Kafka 2.1.

If you are running on older versions of Kafka, you will need to check the compatibility with previous Lenses versions.

Specifically for Lenses SQL processors running inside Kafka Connect, version 2.1 of Kafka Connect is required. In such a case, your Kafka Brokers can be at a different version. The alternative scenario is to run the processors inside Kubernetes.

Note

Whilst Kafka 0.11 should also work, the experience might not be optimal and support will be given on a best-effort basis.

Distributions

Distributions from the following are endorsed:

  1. Apache Kafka
  2. Cloudera
  3. Confluent
  4. HortonWorks

Other distributions and custom builds should work as well but have not been tested.

Lenses is also compatible with several managed services for Kafka such as Azure’s HDInsight.

Connectivity

Your Apache Kafka cluster must be accessible from the host that is running Lenses. Think of it as a typical producer or consumer which should have access to all the brokers in the cluster and the Zookeeper ensemble. Apache Kafka Connect and Schema registries from Hortonworks and Confluent are fully supported; however, their presence is optional.

It is recommended that Lenses is deployed in the same data center as the Apache Kafka cluster it links to.

Enable JMX

In order to make the most out of the monitoring capabilities, JMX metrics need to be enabled for the Kafka Brokers, Zookeepers, Schema Registry, Connect workers.

To enable JMX, the environment variable needs to be set for each of the above components before they are started in the format:

JMX_PORT=[PORT]

where [PORT] is the desired port for each running process.

Additional options should be set in order to access JMX remotely (from a different host) as shown below:

Kafka Brokers

Set JMX_PORT and KAFKA_JMX_OPTS environment variables:

export JMX_PORT=[JMX_PORT]
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=[JMX_PORT]"
Kafka Connect

Set JMX_PORT and KAFKA_JMX_OPTS environment variables:

export JMX_PORT=[JMX_PORT]
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=[JMX_PORT]"
Schema Registry

Set JMX_PORT and SCHEMA_REGISTRY_JMX_OPTS environment variables:

export JMX_PORT=[JMX_PORT]
export SCHEMA_REGISTRY_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=[JMX_PORT]"
Zookeeper

Set JMX_PORT and ZOOKEEPER_SERVER_OPTS environment variables:

export JMX_PORT=[JMX_PORT]
export ZOOKEEPER_SERVER_OPTS="$ZOOKEEPER_SERVER_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=[JMX_PORT]"
Lenses
Lenses also exposes its JMX so you can monitor it. A JMX agent starts automatically for Lenses, metrics are available at port number 9102.

Note

Ensure the ports are reflected correctly in the lenses.conf file. Especially for the brokers, Lenses will autodetect JMX if enabled.

Hardware & OS

The minimum hardware specifications to install and effectively operate Lenses are:

  • 64bit Operating System
  • 4GB of free RAM
  • 500MB of disk space

On the server, a recent Linux 64 bit OS is recommended. RHEL 6.x and CentOS 6.x are partially supported through our Docker image but you cannot run Lenses directly on them.

Open File Descriptors

If you are running on a Linux machine, Lenses requires a slightly larger open file descriptor limit than typical desktop applications. You can check your current limit with the ulimit command:

$ ulimit -n -S # soft limit
$ ulimit -n -H # hard limit

Typically, on Linux, the soft limit is 1024 and can be set up to the hard limit by the user. In order to change the limit, superuser access is needed. A soft limit of 4096 is recommended for Lenses, though 2048 may be enough. To change the soft limit run:

$ ulimit -S -n 4096

Browser

The most recent versions of all major browsers (Chrome, Firefox, Safari) are supported.

TLS Certificates

Lenses can use TLS to serve its API and content as described in Enabling TLS. For this functionality it is required to create a Java keystore file that will provide the certificate and private key pair. For testing purposes you can generate a self signed certificate using the openssl and keytool programs:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365

From the key/cert pair a PKCS12 file should be created:

openssl pkcs12 -export -in ./cert.pem -inkey ./key.pem -certfile ./ca.pem -out lenses.p12

During the export a password is required which is the one that has to be set in the option lenses.ssl.key.password

Finally, the PKCS12 is imported to the keystore.

keytool -importkeystore -srckeystore lenses.p12 -srcstoretype pkcs12 -destkeystore lenses.jks -deststoretype JKS

During the import the password that is used for the keystore has also to be set in lenses.ssl.keystore.password