Lenses Box

Get the Box 

docker pull lensesio/box

What is the Lenses Box? 

Lenses Box is a complete container solution for you to build applications on a localhost Apache Kafka docker.

What’s in the Box? 

Lenses Box contains all components of the Apache Kafka eco-system, CLI tools and synthetic data streams:

• Apache Kafka
• Kafka Connect
• Elasticsearch
• Lenses.io

Get started 

1. To start with Box get your free development license online.

2. Install Docker

Before you get started you will need to install Docker. This Kafka docker container will work even on a low memory 8GB RAM machine.

3. Run Docker

docker run --rm \
    -p 3030:3030 \
    --name=dev \
    --net=host \
    -e EULA="https://dl.lenses.stream/d/?id=CHECK_YOUR_EMAIL_FOR_KEY" \
    lensesio/box

Note that the value of the --name parameter can be anything you want. Just remember to replace dev with your own value in the docker exec commands found in this page. Kafka Connect requires a few minutes to start up since it iterates and loads all the available connectors.

You can also periodically upgrade to the latest versions with:

  $ docker pull lensesio/box

Development experience 

To access the Kafka Docker web user interface, open your browser and navigate to http://localhost:3030

Login with admin / admin.

If interested to learn Kafka or to quick start and create topics, view consumer groups or monitor kafka visit the user guide.

Access the command line 

To access the various Kafka command-line tools, such as the console producer and the console consumer, from a terminal in the Docker container, you should execute the following command:

docker exec -it dev bash
root@fast-data-dev / $ kafka-topics --zookeeper localhost:2181 --list

Alternatively, you can directly execute a Kafka command such as kafka-topics like this:

docker exec -it dev kafka-topics --zookeeper localhost:2181 --list

Kafka Docker advertisement 

The broker in the Kafka docker has a broker id 101 and advertises the listener configuration endpoint to accept client connections.

If you run Docker on macOS or Windows, you may need to find the address of the VM running Docker and export it as the advertised listener address for the broker (On macOS it usually is 192.168.99.100). At the same time, you should give the lensesio/box image access to the VM’s network:

docker run -e EULA="CHECK_YOUR_EMAIL_FOR_KEY" \
           -e ADV_HOST="192.168.99.100" \
           --net=host --name=dev \
           lensesio/box

If you run on Linux you don’t have to set the ADV_HOST but you can do something cool with it. If you set it to be your machine’s IP address you will be able to access Kafka from any clients in your network.

If you decide to run box in the cloud, you (and all your team) will be able to access Kafka from your development machines. Remember to provide the public IP of your server as the kafka advertised host for your producers and consumers to be able to access it.

Kafka Docker JMX 

Kafka JMX metrics are enabled by default. Refer to ports. Once you expose the relevant port ie. -p 9581:9581 you can connect to JMX with

jconsole localhost:9581

Custom hostname 

If you are using docker-machine or setting this up in a Cloud or DOCKER_HOST is a custom IP address such as 192.168.99.100, you will need to use the parameters --net=host -e ADV_HOST=192.168.99.100.

docker run --rm \
    -p 3030:3030 \
    --net=host \
    -e ADV_HOST=192.168.99.100 \
    -e EULA="https://dl.lenses.stream/d/?id=CHECK_YOUR_EMAIL_FOR_KEY" \
    lensesio/box

Docker data persistence 

To persist the Kafka data between multiple executions, provide a name for your Docker instance and do not set the container to be removed automatically (--rm flag). For example:

docker run \
    -p 3030:3030 -e EULA="CHECK_YOUR_EMAIL_FOR_KEY" \
    --name=dev lensesio/box

Once you want to free up resources, just press Control-C. Now you have two options: either remove the Docker:

docker rm dev

Or use it at a later time and continue from where you left off:

docker start -a dev

Port numbers 

Advanced options 

Examples 

The Docker container has been set up to create and produce data to a handful of Kafka topics. The producers (data generators) are enabled by default. In order to disable the examples from being executed set the environment variable -e SAMPLEDATA=0 in the docker run command.

Build SQL streaming flows for Kafka 

Connectors 

The docker ships with a collection of source and sink Kafka Connectors. There is one Kafka Connect Worker with all the connectors pre-set in the classpath so they are ready to be used.

From the UI go to Connectors > New Connector

Follow the instructions of each connector to launch.

SQL Processors 

From the UI go to SQL Processors > New Processor

Here is few examples for SQL Processors for the generated data.

SET defaults.topic.autocreate = true;
INSERT INTO position_reports_Accurate
SELECT STREAM * FROM sea_vessel_position_reports
WHERE Accuracy = true

SET defaults.topic.autocreate = true;
INSERT INTO position_reports_latitude_filter
SELECT STREAM Speed, Heading, Latitude, Longitude, Radio
FROM sea_vessel_position_reports
WHERE Latitude > 58

SET defaults.topic.autocreate = true;
INSERT INTO position_reports_MMSI_large
SELECT STREAM *
FROM sea_vessel_position_reports
WHERE MMSI > 100000

SET defaults.topic.autocreate = true;
INSERT INTO backblaze_smart_result
SELECT STREAM (smart_1_normalized + smart_3_normalized) AS sum1_3_normalized, serial_number
FROM backblaze_smart
WHERE _key.serial_number LIKE 'Z%'

SET defaults.topic.autocreate=true;
SET auto.offset.reset='earliest';
SET commit.interval.ms='10000';

INSERT INTO cc_data_json
  STORE VALUE AS JSON
SELECT STREAM * FROM cc_data;

INSERT INTO cc_payments_json
  STORE VALUE AS JSON
SELECT STREAM * FROM cc_payments;

WITH tableCards AS (
  SELECT TABLE
    number,
    customerFirstName,
    customerLastName,
    blocked
  FROM cc_data_json
);

WITH joined AS (
  SELECT STREAM
    p.merchantId,
  p.creditCardId,
  p.currency
  FROM cc_payments_json AS p JOIN tableCards AS c
  WHERE c.blocked = false
);

INSERT INTO frauds_detection_5
  STORE VALUE AS JSON
SELECT STREAM
  count(*) as attempts
FROM joined
WINDOW BY TUMBLE 5s
GROUP BY joined.merchantId;

FAQ 

How can I run offline? 

Download your key locally and run the command:

LFILE=`cat license.json`
docker run --rm -it -p 3030:3030 -e LICENSE="$LFILE" lensesio/box:latest

How much memory to allocate? 

The container is running multiple services and is recommended to allocate 5GB of RAM to docker (although it can operate with even less than 4GB).

To reduce the memory footprint, it is possible to disable some connectors and shrink the Kafka Connect heap size by applying these options (choose connectors to keep) to the docker run command:

-e DISABLE=azure-documentdb,blockchain,bloomberg,cassandra,coap,druid,elastic,elastic5,ftp,hazelcast,hbase,influxdb,jms,kudu,mongodb,mqtt,redis,rethink,voltdb,yahoo,hdfs,jdbc,elasticsearch,s3,twitter
-e CONNECT_HEAP=512m

Can I connect Box to my Kafka Cluster? 

Lenses Box is optimized for development and includes a single broker Kafka environment. However, you can connect Lenses against your existing setup. Lenses is compatible with all popular Kafka distributions and clouds.

Contact our team to help

Why does my box have topics? 

To make your experience better, we have pre-configured a set of synthetic data generators to get streaming data out of the box. By default the Docker image will launch the data generators; however you can have them off by setting the environment variable -e SAMPLEDATA=0 in the docker run command.

Is box free? 

Yes.

You will need a free key to operate which will need to be renewed after 6 months.

Is box open source? 

Yes. The docker image is open sourced. View on GitHub