Prerequisites

Kafka Versions

Lenses 2.2 supports Kafka from version 0.11 and higher, up to the latest 2.1 version. Kafka 1.0 or higher is strongly recommended.

If you are running on older versions of Kafka, you will need to check the compatibility on previous Lenses versions.

Especially for Lenses SQL processors running inside Kafka Connect, versions 2.0 or 2.1 of Kafka Connect are required. In such case, your Kafka Brokers can be at a different version. The alternative scenario is to run the processors inside Kubernetes.

Distributions

Distributions from the following are endorsed:

  1. Apache Kafka
  2. Cloudera
  3. Confluent
  4. HortonWorks

Other distributions and custom builds should work as well but have not been tested.

Lenses is also compatible with several managed services for Kafka such as Azure’s HDInsight.

Connectivity

Your Apache Kafka cluster must be accessible from the host running Lenses. Think of it as a typical producer or consumer which should have access to all the brokers in the cluster and the Zookeeper ensemble. Apache Kafka Connect and Schema registries from Hortonworks and Confluent are fully supported; however, their presence is optional.

It is recommended that Lenses is deployed in the same data center as the Apache Kafka cluster it links to.

Enable JMX

In order to make the most out of the monitoring capabilities, JMX metrics need to be enabled for the Kafka Brokers, Zookeepers, Schema Registry, Connect workers.

To enable JMX, the environment variable needs to be set for each of the above components before they are started in the format:

JMX_PORT=[PORT]

where [PORT] is the desired port for each running process.

Additional options should be set in order to access JMX remotely (from a different host) as shown below:

Kafka Brokers

Set JMX_PORT and KAFKA_JMX_OPTS environment variables:

export JMX_PORT=[JMX_PORT]
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=[JMX_PORT]"
Kafka Connect

Set JMX_PORT and KAFKA_JMX_OPTS environment variables:

export JMX_PORT=[JMX_PORT]
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=[JMX_PORT]"
Schema Registry

Set JMX_PORT and SCHEMA_REGISTRY_JMX_OPTS environment variables:

export JMX_PORT=[JMX_PORT]
export SCHEMA_REGISTRY_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=[JMX_PORT]"
Zookeeper

Set JMX_PORT and ZOOKEEPER_SERVER_OPTS environment variables:

export JMX_PORT=[JMX_PORT]
export ZOOKEEPER_SERVER_OPTS="$ZOOKEEPER_SERVER_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=[JMX_PORT]"
Lenses
Lenses also exposes its JMX so you can monitor it. A JMX agent starts automatically for Lenses, metrics are available at port 9102.

Note

Ensure the ports are reflected correctly in the lenses.conf file. Especially for the brokers, Lenses will autodetect JMX if enabled.

Hardware & OS

The minimum hardware specifications to install and effectively operate Lenses are:

  • 64bit Operating System
  • 4GB of free RAM
  • 500MB of disk space

On the server, a recent Linux 64 bit OS is recommended. RHEL 6.x and CentOS 6.x are partially supported through our docker image but you can’t run Lenses directly on them.

Open File Descriptors

If you are running on a Linux machine, Lenses requires a slightly larger open file descriptor limit than typical desktop applications. You can check your current limit with the ulimit command:

$ ulimit -n -S # soft limit
$ ulimit -n -H # hard limit

Typically, on Linux, the soft limit is 1024 and can be set up to the hard limit by the user. In order to change the limit, superuser access is needed. A soft limit of 4096 is recommended for Lenses, though 2048 may be enough. To change the soft limit run:

$ ulimit -S -n 4096

Browser

Most recent versions of major browsers (Chrome, Firefox, Safari) are supported.