Cloud¶
Here you can find information on running Lenses in the cloud or with managed Kafka clusters. In addition to the core installation methods (archive and docker), Lenses is offered as a packaged application in various cloud providers.
Cloud Service Discovery¶
In version 2.0 service discovery came into the Lenses docker as a preview feature.
Traditionally, except for the brokers, all other service and jmx endpoints —for Zookeeper, Kafka Connect and Schema Registry— should be explicitly provided to Lenses. This can be cumbersome for a larger cluster or dynamically deployed clusters.
The service discovery feature can help detect the various services endpoints automatically via the metadata services provided in widely used cloud providers, such as Amazon AWS, Google Cloud, Microsoft Azure, DigitalOcean, OpenStack, Aliyun Cloud, Scaleway and SoftLayer. The discovery relies on instances tags to work with.
A list of the available options follow. Options with default values may be omited when the default value corresponds to the correct setup value:
| Variable | Description | Default | Required |
|---|---|---|---|
| SD_CONFIG | Service discovery configuration. Please look
at go-discovery and the examples below
|
— | yes |
| SD_BROKER_FILTER | Filter for Brokers. Please look at
go-discovery and the examples below
|
— | When broker discovery is required |
| SD_BROKER_PORT | Broker Port | 9092 | No |
| SD_BROKER_PROTOCOL | Broker Protocol to use | PLAINTEXT | No |
| SD_ZOOKEEPER_FILTER | Filter for Zookeeper nodes. Please look
at go-discovery and the examples below
|
— | When zookeeper discovery is required |
| SD_ZOOKEEPER_PORT | Zookeeper Port | 2181 | No |
| SD_ZOOKEEPER_JMX_PORT | Zookeeper JMX Port | — | No |
| SD_REGISTRY_FILTER | — | When schema registry
discovery is required
|
|
| SD_REGISTRY_PORT | Schema Registry Port | 8081 | No |
| SD_REGISTRY_JMX_PORT | Schema Registry JMX Port | — | No |
| SD_CONNECT_FILTERS | Comma-separated filters for
connect clusters’ workers.
Please look at go-discovery
and the examples below
|
— | When one or more connect
workers are required
|
| SD_CONNECT_NAMES | Comma-separated names of connect clusters | — | Only if more than one clusters
must be discovered
|
| SD_CONNECT_PORTS | Comma-separated connect workers’ ports | 8083 | No |
| SD_CONNECT_JMX_PORTS | Comma-separated connect workers’ JMX ports | — | No |
| SD_CONNECT_CONFIGS | Comma-separated names of connect
configs topic
|
connect-configs | Only if more than one clusters
must be discovered
|
| SD_CONNECT_OFFSETS | Comma-separated names of connect
offsets topic
|
connect-offsets | only if more than one clusters
must be discovered
|
| SD_CONNECT_STATUSES | Comma-separated names of connect
statuses topic
|
connect-statuses | Only if more than one clusters
must be discovered
|
- Amazon AWS
- You find an example for service discovery for AWS in the following link
- Google Cloud
- You find an example for service discovery for Google Cloud in the following link
- DigitalOcean
Setup for brokers, zookeeper nodes, schema registries and one connect distributed cluster with JMX monitoring, custom ports and SASL_SSL protocol. An read-only API token is needed from DO control panel, in order for service discovery to be able to get a list of running droplets. Private IPv4 Networking should be enabled for the droplets.
SD_CONFIG=provider=digitalocean api_token=[YOUR_API_TOKEN] SD_BROKER_FILTER=region=lon1 tag_name=broker SD_BROKER_PORT=9096 SD_BROKER_PROTOCOL=SASL_SSL SD_ZOOKEEPER_FILTER=region=lon1 tag_name=zookeeper SD_ZOOKEEPER_PORT=10181 SD_ZOOKEEPER_JMX_PORT=10182 SD_REGISTRY_FILTER=region=lon1 tag_name=registry SD_REGISTRY_PORT=19081 SD_REGISTRY_JMX_PORT=19181 SD_CONNECT_FILTERS=region=lon1 tag_name=connect SD_CONNECT_NAMES=production SD_CONNECT_PORTS=19083 SD_CONNECT_JMX_PORTS=19183