4.1
Kafka Connect
Add one or more Kafka Connect cluster by providing a name to distinguish it and all the worker urls (including protocol) and the three backing kafka-connect cluster topics (connect-status, connect-configs and connect-offsets).
Authentication and metrics can be configured as well (for details - check proper tab).
One Kafka Connect cluster with two workers:
The array of worker urls must not be empty.
lenses.kafka.connect.clusters = [
{
name: "changeDataCapture",
urls: [
{ url:"http://CONNECT_HOST_1:8083" },
{ url:"http://CONNECT_HOST_2:8083" }
],
statuses: "connect-status",
configs : "connect-configs",
offsets : "connect-offsets"
# Uncomment and configure accordingly to make this Cluster eligible for deploying SQL processors
# Check https://docs.lenses.io/4.1/configuration/sql/connect/
# ,aes256.key: "0123456789abcdef0123456789abcdef"
}
]
For Aiven you must use the secure https protocol.
lenses.kafka.connect.clusters = [
{
name: "changeDataCapture",
username: "USERNAME",
password: "PASSWORD",
auth: "USER_INFO",
urls: [
// non-empty list of workers urls
],
statuses: "connect-status",
configs: "connect-configs",
offsets: "connect-offsets"
# Uncomment and configure accordingly to make this Cluster eligible for deploying SQL processors
# Check https://docs.lenses.io/4.1/configuration/sql/connect/
# ,aes256.key: "0123456789abcdef0123456789abcdef"
}
]
One Kafka Connect cluster with two workers and JMX metrics.
When defining metrics
objects, particular entries should have the same values of type
field.
Otherwise, the type
of first metrics
object is treatened as the valid one for all metrics.
In principle - the host part of metrics.url
key (“CONNECT_HOST_1” & “CONNECT_HOST_2” in examples below)
should be the same as the worker url
host itself.
lenses.kafka.connect.clusters = [
{
name: "change_data_capture",
urls: [
{ url:"http://CONNECT_HOST_1:8083",
metrics: {
type: "JMX",
url: "CONNECT_HOST_1:9584" # no scheme should be included, just host and port
}
{ url:"http://CONNECT_HOST_2:8083",
metrics: {
type: "JMX",
url: "CONNECT_HOST_2:9584" # no scheme should be included, just host and port
}
],
statuses: "connect-status",
configs : "connect-configs",
offsets : "connect-offsets"
}
]
One Kafka Connect cluster with two workers and authenticated JMX metrics over SSL.
Particular metrics
entries should have the same values of following fields:
type
, ssl
, user
, password
.
Otherwise, the first metrics
object is treatened as the valid one for those keys.
In principle - the host part of metrics.url
key (“CONNECT_HOST_1” & “CONNECT_HOST_2” in examples below)
should be the same as the worker url
host itself.
lenses.kafka.connect.clusters = [
{
name: "change_data_capture",
urls: [
{
url:"http://CONNECT_HOST_1:8083",
metrics: { # Metrics section is (optional)
ssl: true, # Optional (default: false); if used - please make sure the remote
# certificate is accepted by Lenses truststore
user: "admin", # JMX protected by user/pass
password: "admin",
type: "JMX",
url: "CONNECT_HOST_1:9584" # no scheme should be included, just host and port
}
},
{
url:"http://CONNECT_HOST_2:8083",
metrics: { # Metrics section is (optional)
ssl: true, # Optional (default: false); if used - please make sure the remote
# certificate is accepted by Lenses truststore
user: "admin", # JMX protected by user/pass
password: "admin",
type: "JMX",
url: "CONNECT_HOST_2:9584" # no scheme should be included, just host and port
}
}
],
statuses: "connect-status",
configs: "connect-configs",
offsets: "connect-offsets"
}
]
One Kafka Connect cluster with two workers and authenticated Jolokia metrics over SSL.
Particular metrics
entries should have the same values of following fields:
type
, ssl
, user
, password
.
Otherwise, the first metrics
object is treatened as the valid one for those keys.
lenses.kafka.connect.clusters = [
{
name: "change_data_capture",
urls: [
{
url:"http://CONNECT_HOST_1:8083",
metrics: { # Metrics section is (optional)
ssl: true, # Optional (default: false); if used - please make sure the remote
# certificate is accepted by Lenses truststore
user: "", # Optional, the remote HTTP user
password: "", # Optional, the remote HTTP password
type: "JOLOKIAP", # One of 'JOLOKIAG' (for GET requests), 'JOLOKIAP' (for POST requests)
url: "https://host1:9585/jolokia/" # notice the scheme and trailing slash - they are both required for standard Jolokia configuration
}
},
{
url:"http://CONNECT_HOST_2:8083",
metrics: { # Metrics section is (optional)
ssl: true, # Optional (default: false); if used - please make sure the remote
# certificate is accepted by Lenses truststore
user: "", # Optional, the remote HTTP user
password: "", # Optional, the remote HTTP password
type: "JOLOKIAP", # One of 'JOLOKIAG' (for GET requests), 'JOLOKIAP' (for POST requests)
url: "https://host2:9585/jolokia/" # notice the scheme and trailing slash - they are both required for standard Jolokia configuration
}
}
],
statuses: "connect-status",
configs: "connect-configs",
offsets: "connect-offsets"
}
]
Kafka Connect SSL/TLS Http configuration.
Particular metrics
entries should have the same values for these fields:type
, ssl
, user
, password
.
Otherwise, the first metrics
object settings are applied.
In principle - the host part of metrics.url
key (“CONNECT_HOST_1” & “CONNECT_HOST_2” in examples below)
should be the same as the worker url
host itself.
lenses.kafka.connect.clusters = [
{
name: "changeDataCapture",
urls: [
{
url:"http://CONNECT_HOST_1:8083",
metrics: { # Metrics section is (optional)
ssl: true, # SSL - ensure JMX/HTTP certificate is accepted by Lenses truststore
user: "admin", # JMX protected by user/pass
password: "admin",
type: "JOLOKIAP", # One of 'JMX', 'JOLOKIAP' (POST), 'JOLOKIAG' (GET)
url: "https://CONNECT_HOST_1:9584" # JMX port is 9584; Scheme (http or https) is required when using JOLOKIAP and JOLOKIAG !
}
}
],
statuses: "connect-status",
configs: "connect-configs",
offsets: "connect-offsets"
# Uncomment and configure accordingly to make this Cluster eligible for deploying SQL processors
# Check /4.1/configuration/sql/connect/
# ,aes256.key: "0123456789abcdef0123456789abcdef"
}
]
lenses.kafka.connect.ssl.keystore.location = "/path/to/keystore.jks"
lenses.kafka.connect.ssl.keystore.password = "changeit"
lenses.kafka.connect.ssl.key.password = "changeit"
lenses.kafka.connect.ssl.truststore.location = "/path/to/truststore.jks"
lenses.kafka.connect.ssl.truststore.password = "changeit"