Azure HDInsight

Get started with Lenses. Set it up with Docker and connect to an Azure managed Kafka (HDInsight) with the standard configuration.

1. Get your Lenses license key 

Sign up with Lenses and download your license key.

2. Sign up for Azure and create an HDInsight Kafka 

HDInsight for Apache Kafka is a fully managed streaming platform by Azure, deployable in the Azure cloud.

Start by signing up for an Azure account. Follow the in-product instructions to launch HDInsight clusters within the Azure console interface.

Then, gather the connection points of your cluster.

Gather 2 connection points 

Go to Dashboards > Ambari home

1. Kafka endpoints: Go to Kafka > Configs > Kafka Broker > Kafka Broker hosts.

   

2. Zookeeper endpoints: Go to Zookeeper > Configs > Zookeeper Server > Zookeeper Server hosts.

   

3. Get Docker 

You will use docker-compose to spin up Lenses.

Start by signing up with Docker. Then install docker-compose.

4. Setup Lenses docker-compose 

Use this docker-compose.yml file to spin up Lenses and connect to your Azure HDInsight Kafka.

Just 2 steps:

1. Fill in the connection details:
   1. Kafka endpoint
   2. Zookeeper endpoint
2. Share the authentication files with your docker. Copy these files in the same directory as your docker-compose.yml file and reference them:
   1. Your Lenses license key license.json

The file:

version: '3'
services:
  lenses:
    image: lensesio/lenses
    environment:
      # Admin user credentials
      LENSES_SECURITY_USER: admin
      LENSES_SECURITY_PASSWORD: admin
      LENSES_PORT: 8080
      
      # === Set your values in this section ===
      
      # ===Step 1=== Put your HDInsight Kafka endpoints here
      # e.g. 
      # LENSES_KAFKA_BROKERS: "PLAINTEXT://[wn0-lenses.kg3i1hefhdzuhl5mvhdd00n3nd.zx.internal.cloudapp.net]:9092, PLAINTEXT://wn1-lenses.kg3i1hefhdzuhl5mvhdd00n3nd.zx.internal.cloudapp.net:9092"
      # Kafka Endpoint
      LENSES_KAFKA_BROKERS: "PLAINTEXT://[BROKER_HOST_1]:9092, PLAINTEXT://[BROKER_HOST_2]:9092,…"
      
      # ===Step 2=== Put your HDInsight Zookeeper endpoints here
      LENSES_ZOOKEEPER_HOSTS: |
        [
          { url: "[ZK_HOST_1]:2181" },
          { url: "[ZK_HOST_2]:2181" },
          { url: "[ZK_HOST_3]:2181" }
        ]        
      
      # HDInsight JMX metrics port for Kafka brokers
      LENSES_KAFKA_METRICS_DEFAULT_PORT: 9999        
      
    ports:
      - 8080:8080
    volumes:
      # ===Step 3=== Share your authentication files here
      # Map the Lenses license key file here to activate Lenses
      - ./license.json:/license.json

4. Start Lenses up 

docker-compose up

5. Login 

Open your browser and go to:

http://localhost:8080

Login as admin:

User:     admin
Password: admin

6. Eureka! 

Next steps 

• Learn how to use Lenses.
  

Limitations 

Currently, Azure HDInsight doesn’t provide managed versions of these services:

• Schema Registry
• Kafka Connect

Without these services these Lenses features are not available: