GCP¶

Lenses can be deployed in GCP with your own Apache Kafka cluster with a Deployment manager template. The fields which you need to fill in are:

Values Configuration¶

Field	Description	Type	Default	Required
`machineType`	The type of the machine which lenses run. Recommended >= n1-standard-2	string	n1-standard-2	no
`network`	The network in which Lenses.io will be deployed	string	default	no
`lensesLicense`	The Lenses license. You can download here https://lenses.io/downloads/	string		yes
`lensesPort`	The port which Lenses will run and be available	int	9991	no
`lensesPassword`	The admin password of Lenses	string	admin	no
`zone`	The GCP zone to run the VM	string		yes
`sourceRanges`	Source ranges for Firewall configuration	string	0.0.0.0/0	yes
`kafkaBrokers`	A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. Add just a few broker addresses here and Lenses will bootstrap and discover the full cluster membership (which may change dynamically). This list should be in the form `"host1:port1,host2:port2,host3:port3"`	string		yes
`zookeeper`	A list of all the zookeeper nodes	array		no
`connect`	Defines the Kafka connect clusters	array		no
`schemaRegistry`	A list of Schema Registry nodes	array		no

Deployment Manager Templates¶

There is one template which you can use to deploy Lenses in GCP. Based on Hardware & OS requirements for Lenses, we recommend to start with n1-standard-2 instances or any other instance type with >= 7GB.

Container Optimized Deployment¶

Container-Optimized OS is an operating system image for your Compute Engine VMs that is optimized for running Docker containers. With Container-Optimized VM, you can bring up Lenses running with Docker container on Google Cloud Platform quickly, efficiently, and securely.

The Container Optimized Deployment for Lenses supports also service discovery of Kafka Brokers, Zookeeper, Schema Registry and Connect based on the provided zone. Use the config.yaml to override the values for your configuration.

imports:
- path: lenses.jinja

resources:
- name: lenses
  type: lenses.jinja
  properties:
    zone: europe-west3-a
    sourceRanges: "0.0.0.0/0"
    machineType: n1-standard-2
    kafkaBrokers: "PLAINTEXT://<YOUR-KAFKA-BROKER-IP>:9092"
    lensesLicense: |
    zookeeper: |
      [
        {url:"zookeeper.1.url:2181", jmx:"zookeeper.1.url:9585"},
        {url:"zookeeper.2.url:2181", jmx:"zookeeper.2.url:9585"}
      ]
    schemaRegistry: |
      [
        {url:"http://schema.registry.1.url:8081",jmx:"schema.registry.1.url:9582"},
        {url:"http://schema.registry.2.url:8081",jmx:"schema.registry.2.url:9582"}
      ]
    connect: |
      [
        {
          name:"data_science",
          urls: [
            {url:"http://connect.worker.1.url:8083",jmx:"connect.worker.1.url:9584"},
            {url:"http://connect.worker.2.url:8083",jmx:"connect.worker.2.url:9584"}
          ],
          statuses:"connect-statuses-cluster-a",
          configs:"connect-configs-cluster-a",
          offsets:"connect-offsets-cluster-a"
        }
      ]

You can find in Github Repo the available template. You can deploy it with the following command:

gcloud deployment-manager deployments create lenses --config config.yaml

Service Discovery¶

Setup for brokers, zookeeper nodes, schema registries and one connect distributed cluster with JMX monitoring and default ports. left at default values. Lenses VM should have the scope https://www.googleapis.com/auth/compute.readonly.

SD_CONFIG=provider=gce zone_pattern=europe-west1.*

SD_BROKER_FILTER=tag_value=broker

SD_ZOOKEEPER_FILTER=tag_value=zookeeper
SD_ZOOKEEPER_JMX_PORT=9585

SD_REGISTRY_FILTER=tag_value=schema-registry
SD_REGISTRY_JMX_PORT=9582

SD_CONNECT_FILTERS=tag_value=connect-worker-testing,tag_value=connect-worker-production
SD_CONNECT_NAMES=testing,production
SD_CONNECT_STATUSES=connect-statuses-testing,connect-statuses-production
SD_CONNECT_CONFIGS=connect-configs-testing,connect-configs-production
SD_CONNECT_OFFSETS=connect-offsets-testing,connect-offsets-production
SD_CONNECT_JMX_PORTS=9584