Kafka cloud service discovery

Service discovery enables automatic configuration and detection on endpoints via cloud providers metadata services. Service discovery is available on:

AWS
Google
Azure
DigitalOcean
OpenStack
Scaleway
Softlayer

Some examples are:

Setup for Brokers, ZK, schema registries and one kafka connect cluster with all the values (ports, connect topics, protocol) left at default. The Lenses VM will need the IAM permission ec2:DescribeInstances to auto-discover configuration.

SD_CONFIG=provider=aws region=eu-central-1 addr_type=public_v4
SD_BROKER_FILTER=tag_key=Name tag_value=*broker*
SD_ZOOKEEPER_FILTER=tag_key=Name tag_value=*zookeeper*
SD_REGISTRY_FILTER=tag_key=Name tag_value=*worker*
SD_CONNECT_FILTERS=tag_key=Name tag_value=*worker*

On the above example, the Schema Registry runs in the same instances as Connect and will work out of the box if you use Confluent’s AWS templates to deploy your cluster.

Setup Brokers, ZK, schema registries and one kafka connect cluster with all the values (default ports, connect topics, protocol) set at default. The Lenses VM will need the scope https://www.googleapis.com/auth/compute.readonly.

SD_CONFIG=provider=gce zone_pattern=europe-west1.*

SD_BROKER_FILTER=tag_value=broker

SD_ZOOKEEPER_FILTER=tag_value=zookeeper
SD_ZOOKEEPER_JMX_PORT=9585

SD_REGISTRY_FILTER=tag_value=schema-registry
SD_REGISTRY_JMX_PORT=9582

SD_CONNECT_FILTERS=tag_value=connect-worker-testing,tag_value=connect-worker-production
SD_CONNECT_NAMES=testing,production
SD_CONNECT_STATUSES=connect-statuses-testing,connect-statuses-production
SD_CONNECT_CONFIGS=connect-configs-testing,connect-configs-production
SD_CONNECT_OFFSETS=connect-offsets-testing,connect-offsets-production
SD_CONNECT_JMX_PORTS=9584

Setup Brokers, ZK, schema registries and one kafka connect cluster with JMX monitoring, custom ports and SASL_SSL protocol. A read-only API token is required from DO control panel, in order for service discovery to discover all droplets. Private IPv4 Networking should be enabled for the droplets.

SD_CONFIG=provider=digitalocean api_token=[YOUR_API_TOKEN]

SD_BROKER_FILTER=region=lon1 tag_name=broker
SD_BROKER_PORT=9096
SD_BROKER_PROTOCOL=SASL_SSL

SD_ZOOKEEPER_FILTER=region=lon1 tag_name=zookeeper
SD_ZOOKEEPER_PORT=10181
SD_ZOOKEEPER_JMX_PORT=10182

SD_REGISTRY_FILTER=region=lon1 tag_name=registry
SD_REGISTRY_PORT=19081
SD_REGISTRY_JMX_PORT=19181

SD_CONNECT_FILTERS=region=lon1 tag_name=connect
SD_CONNECT_NAMES=production
SD_CONNECT_PORTS=19083
SD_CONNECT_JMX_PORTS=19183

Complete list of options

Please look at go-discovery and the examples below

Variable	Description	Default	Required
SD_CONFIG	Service discovery configuration	-	yes
SD_BROKER_FILTER	Filter for Broker discovery	-	yes
SD_BROKER_PORT	Broker Port	9092	no
SD_BROKER_PROTOCOL	Broker Protocol	PLAINTEXT	no
SD_ZOOKEEPER_FILTER	Filter for Zookeeper nodes	-	no
SD_ZOOKEEPER_PORT	Zookeeper Port	2181	no
SD_ZOOKEEPER_JMX_PORT	Zookeeper JMX Port	—	no
SD_REGISTRY_FILTER	Filter for Schema Registries	-	no
SD_REGISTRY_PORT	Schema Registry Port	8081	no
SD_REGISTRY_JMX_PORT	Schema Registry JMX Port	—	no

For Kafka Connect auto-discovery use comma separated list when more than one cluster is in place:

Variable	Description	Default	Required
SD_CONNECT_FILTERS	Filter for workers	—	no
SD_CONNECT_NAMES	Names of clusters	—	when > 1 cluster
SD_CONNECT_PORTS	Worker ports	8083	no
SD_CONNECT_JMX_PORTS	Worker JMX ports		no
SD_CONNECT_CONFIGS	Names of config topic	connect-configs	no
SD_CONNECT_OFFSETS	Names of offsets topic	connect-offsets	no
SD_CONNECT_STATUSES	Names of statuses topic	connect-statuses	no