IBM

Lenses can be deployed in AWS with your own Apache Kafka or IBM Managed Streaming for Apache Kafka (Event Streams) with a Terraform module.

Some features are not available with Event Streams, namely ZooKeeper and the JMX ports on the Kafka brokers - these are not required to run Lenses.

Quickstart

To run Lenses in the Cloud go to our Github page, and checkout the repo

git clone https://github.com/lensesio/ibm-cloud-schematic

Create an SSH Key

Get a Lenses license

Run the command

make

You should be prompted for several environment variables to configure Terraform and the virtual machine will be created.

For troubleshooting and more instructions See the README’s in the project directories.

Event Streams Service Credentials

The lenses_jaas_password, lenses_jaas_username and lenses_kafka_brokers are gleened from the Event streams service credentials.

The following table maps the Event Streams service credential to the Terraform parameter.

Service Credential Terraform Parameter
user lenses_jaas_password
pasword lenses_jaas_password
kafka_brokers_sasl lenses_kafka_brokers
Example Service Credentials

Terraform Variables

Required fields

Below are descriptions of the variables required to run Lenses. Some of them have defaults, the others need to be prepopulated in order for the deployment to complete successfully.

Field Description Type Default Required
cores
The number of CPU cores that you want to allocate.
 string 2 yes
datacenter The datacenter in which you want to provision the instance. string lon02 yes
domain The domain for the computing instance. string n/a yes
hostname
An optional the hostname for the computing instance. Defaults to lenses.
 string lenses yes
hourly_billing
The billing type for the instance. When set to true the computing instance is billed on hourly usage.
Otherwise the instance is billed on a monthly basis. The default value is true.
 bool true yes
iaas_classic_api_key IBM Cloud classic infrastructure API key. string n/a yes
iaas_classic_username IBM Cloud classic infrastructure user name. string n/a yes
lenses_jaas_password The event streams service credentials password. string n/a yes
lenses_jaas_username The event streams service credentials username. string n/a yes
lenses_kafka_brokers
A string of brokers formed from the service credentials kafka_brokers_sasl field.
Of the form SASL_SSL://broker0_url:9093,SASL_SSL://broker1_url:9093..
 string n/a yes
lenses_webgui_password
A password to log into the instance with. Default is admin.
 string admin yes
lenses_webgui_username
A username to log into the instance with. Default is admin.
 string admin yes
lenses_release The major release of lenses of the form x.0 string 3.0 yes
lenses_version The major/minor/patch number for lenses release. Of the form x.x.x string 3.0.6 yes
lenses_webgui_password A password to log into the instance with. Default is admin. string admin yes
lenses_webgui_username A username to log into the instance with. Default is admin. string admin yes
memory
The amount of memory expressed in megabytes that you want to allocate.
 string 4096 yes
os_reference_code
The operating system reference code that is used to provision the computing instance.
 string CENTOS_7_64 yes
private_network_only
When set to true a compute instance only has access to the private network. The default value is false.
 bool false yes
private_ssh_path The path to the private SSH key for the precedding public key (required for provisioning) string ~/.ssh/id_rsa yes
public_key The SSH public key string to install on the computing instance when the instance provisions. string n/a` yes
region IBM Cloud region string n/a yes

Optional fields

Optional values if you want to use the Log Analysis service.

Field Description Type Default Required
logdna_analysis_key
Key to allow logdna-agent access to write logs to IBM Log Analysis
 string n/a no
logdna_region
logdna-agent Region for IBM Log Analysis
 string eu-gb no

Manual Configuration

If you want to configure the instance yourself follow the instructions below. It’s recommended to use CentOS to install Lenses, however Ubuntu is a popular linux distro which the instructions below have been written for.

Create a Virtual Instance

See the Hardware & OS requirements for Lenses.

We used a server with the following characteristics:

  • 4 vCPU
  • 8 GB RAM
  • 25 GB Disk
  • Ubuntu Linux 18.04 LTS Bionic Beaver Minimal Install (64 bit)
Example Server Configuration

Note

When creating the instance we found that we had no internet access. This turned out to be a problem with the /etc/resolv.conf settings. See this article for more information. https://bugs.launchpad.net/ubuntu/+source/resolvconf/+bug/1817903 Removing the symlink and changing this to nameserver 8.8.8.8 fixed the problem. Still having problems? - double check your security groups.

Install Prerequisites

For this guide we used the archive of lenses. Download the latest version of lenses, from https://archive.lenses.io/lenses/. Or run the script below to download Lenses and initialize the config.

export LENSES_BASE_VERSION=<base-version>
export LENSES_VERSION=<specific-version>
wget -qO- https://archive.lenses.io/lenses/$LENSES_BASE_VERSION/lenses-$LENSES_VERSION-linux64.tar.gz | tar -xvz -C /opt
export LENSES_HOME=/opt/lenses
mkdir -p $LENSES_HOME/data/storage
touch $LENSES_HOME/lenses.conf
touch $LENSES_HOME/jaas.conf
touch $LENSES_HOME/license.json
touch $LENSES_HOME/security.conf
export LENSES_OPTS="-Djava.security.auth.login.config=$LENSES_HOME/jaas.conf"
cd $LENSES_HOME

A minimal config to run Lenses with event streams requires 4 files:

  1. lenses.conf
  2. license.json
  3. jaas.conf
  4. security.conf

Running the above script will have created these 4 files for you. Now simply populate them with the following content:

# Set the ip:port for Lenses to bind to
lenses.ip = 0.0.0.0
lenses.port = 9991

# License file allowing connecting to up to N brokers
lenses.license.file = "/opt/lenses/license.json"

# Directory where Lenses stores local storage.
# If omitted it will create a directory named 'storage' under the current directory.
# Write access is needed as well as surviving between upgrades.
lenses.storage.directory = "/opt/lenses/data/storage"

# Set up infrastructure end-points

# The more brokers you can add here, the better
lenses.kafka.brokers = "SASL_SSL://BROKER_HOST_NAME:9093"

# consumer settings
lenses.kafka.settings.consumer.security.protocol="SASL_SSL"
lenses.kafka.settings.consumer.sasl.mechanism="PLAIN"
lenses.kafka.settings.consumer.ssl.protocol="TLSv1.2"

# producer settings
lenses.kafka.settings.producer.security.protocol="SASL_SSL"
lenses.kafka.settings.producer.sasl.mechanism="PLAIN"
lenses.kafka.settings.producer.ssl.protocol="TLSv1.2"

# Processor Mode & State dir options
lenses.sql.execution.mode = "IN_PROC"
lenses.sql.state.dir = "/tmp/lenses-sql-kstream-state"

Substitute the lenses.kafka.brokers configuration option with value the kafka_brokers_sasl array values, from the service credentials. Make sure to concatenate the values with a comma, and prepend them with SASL_SSL://.

# The more brokers you can add here, the better
lenses.kafka.brokers = "SASL_SSL://BROKER_HOST_NAME_1:9093,SASL_SSL://BROKER_HOST_NAME_2:9093"

Populate this file with the JSON blob from your license.

{
    "source": "Lenses.io Ltd",
    "clientId": "CLIENT_ID",
    "details": "Lenses",
    "key": "KEY"
}

Substitute the USERNAME/PASSWORD variables with the respective options supplied in the Event Streams service credentials. (See the previous screenshot on service credentials)

KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="USERNAME"
  password="PASSWORD";
};

You can change the default Lenses username and password with the following configuration:

lenses.security.user = "admin"
lenses.security.password = "admin"

Run the Lenses binary to start the service

nohup /opt/lenses/bin/lenses &

Finally navigate to the IP address of the virtual server on port 9991 with your browser, and log into lenses.

References