Kafka Connect

Lenses uses Kafka Connect to manage connectors.

Multiple Kafka Connect clusters might be used - a separate Connection should be created for each. Once a Kafka Connect connection is defined, Lenses displays all its workers and their statuses (with metrics, if configured so) in Workspace->Services screen.

Users need explicit permission via their groups to see a Connect cluster. In order to view, create, or manage a connector, they further need permissions to the topics it accesses.

The name of a Kafka Connect Connections may only contain alphanumeric characters ([A-Za-z0-9]) and dashes (-). Valid examples would be dev, Prod1, SQLCluster,Prod-1, SQL-Team-Awesome.

Adding a connection 

To add a connection, go to:

Admin ➡ Connections

Select the New Connection button and select Kafka Connect.

Enter the details.

Modifying a connection 

To edit a connection, go to:

Admin ➡ Connections

select the Connect connection and in actions select edit.:

Metrics 

Metrics, if configured, are shown in each connector’s page and in the topology view. If some workers are ommited, the metrics will be incomplete, each worker only exports its own metrics.

Connection details 

Like other core services, Kafka Connect is managed via Connections. See more about managing connections here.

Multiple connections allowed: YES

Deletion allowed: YES

When managing connections other than with GUI, such information are relevant for Kafka Connect connection:

Template name: KafkaConnect

Provision yaml examples 

The easiest way to manage Connections is via Lenses GUI under their respective pages, however it is also possible to do it directly via API, Helm or Lenses CLI. In such case, some connection type-specific values have to be used. Here are few examples of such configuration in YAML format.

• Find out more about managing Kafka Connect Connections via API
• Find out more about managing Connections via Lenses Provisioning
• Find out more about installing Lenses via Helm

Simple configuration, with JMX metrics 

The URLs (workers) should always have a scheme defined (http:// or https://).

This example uses an optional AES-256 key. The key decodes values encoded with AES-256 to enable passing encrypted values to connectors. It is only needed if your cluster uses AES-256 Decryption plugin.

your-cluster-name:
  tags: ["tag1"]
  templateName: KafkaConnect
  configurationObject:
    workers:
      - http://my-kc.worker1:8083
      - http://my-kc.worker2:8083
    aes256Key: PasswordPasswordPasswordPassword
    # all metrics properties are optional
    metricsPort: 9581
    metricsType: JMX
    metricsSsl: false

Misc metrics configurations 

Find more about multiple options of configuring services’ metrics (like secured JMX, Jolokia, etc) under Services Metrics

Basic authentication 

For Basic Authentication, define username and password properties.

your-cluster-name:
  tags: ["tag1"]
  templateName: KafkaConnect
  configurationObject:
    workers:
      - http://my-kc.worker1:8083
      - http://my-kc.worker2:8083
    username: my-username
    password: my-password

TLS with custom truststore 

A custom truststore is needed when the Kafka Connect workers are served over TLS (encryption-in-transit) and their certificates are not signed by a trusted CA.

your-cluster-name:
  tags: ["tag1"]
  templateName: KafkaConnect
  configurationObject:
    workers:
      - https://my-kc.worker1:8083
      - https://my-kc.worker2:8083
    sslTruststore:
      fileRef:
        filePath: /path/to/my/truststore.jks
    sslTruststorePassword: myPassword

TLS with client authentication 

A custom truststore might be necessary too (see above).

your-cluster-name:
  tags: ["tag1"]
  templateName: KafkaConnect
  configurationObject:
    workers:
      - https://my-kc.worker1:8083
      - https://my-kc.worker2:8083
    sslKeystore:
      fileRef:
        filePath: /path/to/my/keystore.jks
    sslKeystorePassword: keystorePassword
    sslKeyPassword: keyPassword

TLS with Basic Authentication 

As above - a custom truststore is needed when the Kafka Connect workers are served over TLS (encryption-in-transit) and their certificates are not signed by a trusted CA.

your-cluster-name:
  tags: ["tag1"]
  templateName: KafkaConnect
  configurationObject:
    workers:
      - https://my-kc.worker1:8083
      - https://my-kc.worker2:8083
    sslTruststore:
      fileRef:
        filePath: /path/to/my/truststore.jks
    sslTruststorePassword: myPassword
    username: my-username
    password: my-password