Cloud

Cloud Service Discovery

In version 2.0 service discovery came into the Lenses docker as a preview feature.

Traditionally, except for the brokers, all other service and jmx endpoints —for Zookeeper, Kafka Connect and Schema Registry— should be explicitly provided to Lenses. This can be cumbersome for a larger cluster or dynamically deployed clusters.

The service discovery feature can help detect the various services endpoints automatically via the metadata services provided in widely used cloud providers, such as Amazon AWS, Google Cloud, Microsoft Azure, DigitalOcean, OpenStack, Aliyun Cloud, Scaleway and SoftLayer. The discovery relies on instances tags to work with.

A list of the available options follow. Options with default values may be omited when the default value corresponds to the correct setup value:

Variable Description Default Required
SD_CONFIG
Service discovery configuration. Please look
at go-discovery and the examples below
 — yes
SD_BROKER_FILTER
Filter for Brokers. Please look at
go-discovery and the examples below
 — When broker discovery is required
SD_BROKER_PORT Broker Port 9092 No
SD_BROKER_PROTOCOL Broker Protocol to use PLAINTEXT No
SD_ZOOKEEPER_FILTER
Filter for Zookeeper nodes. Please look
at go-discovery and the examples below
 — When zookeeper discovery is required
SD_ZOOKEEPER_PORT Zookeeper Port 2181 No
SD_ZOOKEEPER_JMX_PORT Zookeeper JMX Port — No
SD_REGISTRY_FILTER
Filter for Schema Registries. Please look at
go-discovery and
the examples below
 —
When schema registry
discovery is required
SD_REGISTRY_PORT Schema Registry Port 8081 No
SD_REGISTRY_JMX_PORT Schema Registry JMX Port — No
SD_CONNECT_FILTERS
Comma-separated filters for
connect clusters’ workers.
Please look at go-discovery
and the examples below
 —
When one or more connect
workers are required
SD_CONNECT_NAMES Comma-separated names of connect clusters —
Only if more than one clusters
must be discovered
SD_CONNECT_PORTS Comma-separated connect workers’ ports 8083 No
SD_CONNECT_JMX_PORTS Comma-separated connect workers’ JMX ports — No
SD_CONNECT_CONFIGS
Comma-separated names of connect
configs topic
 connect-configs
Only if more than one clusters
must be discovered
SD_CONNECT_OFFSETS
Comma-separated names of connect
offsets topic
 connect-offsets
only if more than one clusters
must be discovered
SD_CONNECT_STATUSES
Comma-separated names of connect
statuses topic
 connect-statuses
Only if more than one clusters
must be discovered

Examples of service discovery configuration in various clouds follow.

Amazon AWS

Setup for brokers, zookeeper nodes, schema registries and one connect distributed cluster without JMX and everything (ports, connect topics, protocol) left at default values. Lenses VM should have the IAM permission ec2:DescribeInstances. The Schema Registry runs in the same instances as Connect. This example would actually work if you used Confluent’s AWS templates to deploy your cluster.

SD_CONFIG=provider=aws region=eu-central-1 addr_type=public_v4
SD_BROKER_FILTER=tag_key=Name tag_value=*broker*
SD_ZOOKEEPER_FILTER=tag_key=Name tag_value=*zookeeper*
SD_REGISTRY_FILTER=tag_key=Name tag_value=*worker*
SD_CONNECT_FILTERS=tag_key=Name tag_value=*worker*
Google Cloud

Setup for brokers, zookeeper nodes, schema registries and one connect distributed cluster with JMX monitoring and default ports. left at default values. Lenses VM should have the scope https://www.googleapis.com/auth/compute.readonly.

SD_CONFIG=provider=gce zone_pattern=europe-west1.*

SD_BROKER_FILTER=tag_value=broker

SD_ZOOKEEPER_FILTER=tag_value=zookeeper
SD_ZOOKEEPER_JMX_PORT=9585

SD_REGISTRY_FILTER=tag_value=schema-registry
SD_REGISTRY_JMX_PORT=9582

SD_CONNECT_FILTERS=tag_value=connect-worker-testing,tag_value=connect-worker-production
SD_CONNECT_NAMES=testing,production
SD_CONNECT_STATUSES=connect-statuses-testing,connect-statuses-production
SD_CONNECT_CONFIGS=connect-configs-testing,connect-configs-production
SD_CONNECT_OFFSETS=connect-offsets-testing,connect-offsets-production
SD_CONNECT_JMX_PORTS=9584
DigitalOcean

Setup for brokers, zookeeper nodes, schema registries and one connect distributed cluster with JMX monitoring, custom ports and SASL_SSL protocol. An read-only API token is needed from DO control panel, in order for service discovery to be able to get a list of running droplets. Private IPv4 Networking should be enabled for the droplets.

SD_CONFIG=provider=digitalocean api_token=[YOUR_API_TOKEN]

SD_BROKER_FILTER=region=lon1 tag_name=broker
SD_BROKER_PORT=9096
SD_BROKER_PROTOCOL=SASL_SSL

SD_ZOOKEEPER_FILTER=region=lon1 tag_name=zookeeper
SD_ZOOKEEPER_PORT=10181
SD_ZOOKEEPER_JMX_PORT=10182

SD_REGISTRY_FILTER=region=lon1 tag_name=registry
SD_REGISTRY_PORT=19081
SD_REGISTRY_JMX_PORT=19181

SD_CONNECT_FILTERS=region=lon1 tag_name=connect
SD_CONNECT_NAMES=production
SD_CONNECT_PORTS=19083
SD_CONNECT_JMX_PORTS=19183