AWS

Lenses.io is an advanced Amazon MSK integration offering data observability, productivity, monitoring, security and governance for Apache Kafka and event/streaming applications.

MSK Kafka and Lenses EC2 - MSK (Use hourly)
MSK Kafka and Lenses EC2 - MSK (Use license)

Learn more 

Read bellow to learn more about

AWS Marketplace 

Select CloudFormation Template, Lenses EC2 and your region

MSK CloudFormation template

2) Choose Launch CloudFormation

MSK launch Lenses action

3) Continue with the default options for creating the stack in the AWS wizard.

Fill in the parameters at Specify stack details.

  • Deployment Here the EC2 instance size and password for the Lenses admin user are set. A t2.large instance size is recommended;
  • Network Configuration This section controls the network settings of the Lenses EC2 instance. The ingress allows access to the Lenses UI only from particular IP addresses;
  • MSK Set the Security Group ID to that of your MSK cluster. A rule will be added to it so that Lenses can communicate with your cluster. You can find the ID by navigating in the AWS console to your MSK cluster and then under Properties -> Networking settings;
  • Monitoring Optionally produce the Lenses logs to CloudWatch;
  • Storage Lenses can stores its state in a database locally on the EC2 instance’s disk or in a PostgreSQL database. Local storage is a development/quickstart option and not suitable for production use. It is advised to use a postgres database for smoother upgrades.

4) Review the stack:

MSK Lenses installation review

5) Accept the terms and conditions and create the stack:

MSK Lenses ack

6) Once the stack has deployed, go to the Output tab and click on the FQDN link. If there are no outputs listed you might need to press the refresh button.

MSK stack FQDN link

7) Login to Lenses with admin and the password value you have submitted on parameter LensesAdminPassword.

Template policies 

The template enables the following policies:

  • logs:CreateLogGroup
  • logs:CreateLogStream
  • logs:PutLogEvents
  • logs:PutLogEvents

Persistence Storage options 

In-order to do in-place Lenses version upgrades, by downloading Linux binary from Lenses Archive, and you are using storage mode local it is suggested first to do an H2 database backup as in the following article

Connect with SSH to EC2 Instance 

In version 5.0.0 and onwards port 22 is no longer allowed as an Inbound rule, on the Lenses security group. This is in sync with AWS Marketplace security recommendations, since most users would leave the default allow rule of 0.0.0.0/0, which is a known security risk.

However, to be able to connect with SSH to the instance, you can apply the following workaround step-by-step guide, that utilizes EC2 Instance Connect browser feature:

1) Find your cloudformation stack and open the stack resources menu

Cloudformation Stack
Stack Resources

2) From there you can find the Lenses Instance details and on the Security tab, go to the Lenses Security group to Edit inbound rules.

EC2 Details
Lenses Security Group

3) Add a temporary SSH allow rule with type SSH, for All Ipv4 addresses and Save. This is a pre-requisite for EC2 Instance Connect feature.

Add 22 Rule All
Rule Applied

4) Go back to instance details and hit the Connect button. It will take you to the EC2 Instance Connect screen, to SSH via the browser. Leave parameters user root as is and hit Connect

Use the browser terminal to edit ~/.ssh/authorized_keys file.

EC2 Instance Connect
EC2 Instance Connect Terminal

5) Assuming you have an SSH keypair, copy the public key content to ~/.ssh/authorized_keys. EC2 Instance Connect terminal, uses Ctrl+V to paste copied content from outside the browser screen (such as clipboard). Save the file when done.

You can generate a new secure keypair using ssh-keygen -t rsa -b 4096 command.

Public Key setup

6) Now, after adding a public key, you no longer need EC2 Instance Connect so you can restrict SSH either to a specific IP address, e.g. a VPN server address or only your own public IP, using option My IP on the previous SSH rule.

Restrict SSH Rule
Restrict SSH Applied

7) You can now connect from your local environment to the Lenses Instance, using the private key as root user. Remember to revoke SSH access when done.

SSH Login

IAM Support 

Lenses supports connection to MSK brokers via IAM. If Lenses is deployed on an EC2 instance it will use the default credential chain loader to authentic and connect to MSK.

Summary 

Lenses adds to Amazon MSK a secure User Interface with DataOps capabilities for:

  • Data discovery
  • Data security
  • Data governance
  • Data monitoring
  • Data observability
  • Data alerts

Installation methods 

1. Secure AWS installation
2. AWS marketplace (Hourly usage)
3. AWS marketplace (BYOL - Bring Your own license) - get a trial license
4. AWS EDP Private Offer contact us for an AWS EDP offer.

Help video