Lenses integrates with Kafka Connect to create, manage and monitor connectors via multiple connect clusters. In this guide you will teach you how to spin up and manage Kafka Connectors and monitor the generated flows.
Kafka Connect is one of the core APIs of Kafka ecosystem. Connectors are reusable components to bring data in and out of a Kafka cluster.
When using Kafka Connect clusters you can authorise clusters per group. Select the approriate clusters per group to make it available to the users.
You can only use data within your data namespaces. If a topic is outside your namespace, you won’t be able to create a connector to either read or write to this topic.
You can connect Lenses to one or multiple Kafka Connect clusters by configuring the cluster connection to Lenses config. For every cluster you will give an alias name, which will be used in Lenses, to help you navigate through the clusters where required.
To use Kafka Connectors, the cluster as well as the connector plugins need to be available. Connectors are made available via the cluster classpath and you can optionally add metadata to Lenses for better management.
When multiple clusters are available, you can use group management to authorize users for each cluster and control visibility.
Once a cluster is made available in Lenses, all running connectors will be listed under the connectors list. You can have a quick overview per cluster and filter connector accordingly.
When a connector has a failure or is not running you will be able to preview it from the listing and drill down to connector details to view more details about the error as reported by the connector:
The stacktrace exception can be returned by the connector:
Relevant metrics are also available for each task on your connector.
You can start/stop the connector via the detailed view or edit the existing configuration. Lenses uses the native connect interfaces to apply the changes.
To create a new connector navigate to the Connectors list from the sidebar and select New Connector. You will be prompt with a list of connector plugins available in each cluster. Switch the cluster to have a look at the different connectors available:
The connector plugins are listed based on sources (bring data in Kafka) and sinks (bring data out of Kafka). Select the desired plugin and click to fill in the configuration details for the connector.
The mandatory fields will be prompted, you can select the optional fields to appear. The validation is running so that add all the required configurations:
For every sink connector the underlying consumer can be tracked. From there, you can manage the offsets ( ie. reset offsets or skip ) and also from the alert rules you can add consumer lag alerts for the relevant consumer. To edit the offsets your connector, therefore your consumer, should not be running ( aka committing offsets )
Any Kafka Connector that is following the Connect API standards, even if it is your custom connector, is compatible with Lenses, as long as it’s available in the cluster’s classpath and exposed via connect’s apis.
For better visibility, we advice to configure your connector’s details in Lenses so you can view its metadata.
Your connectors instances are automatically plotted to the topology view of Lenses. From there you can see connectors and how they are linked with the rest of your streaming pipeline. Your topological view is also determined by your permissions.
If you want to have better visibility of a 3rd party connector you can optionally enhance the information about it:
lenses { ... connectors.info = [ { class.name = "The connector full classpath" name = "The name which will be presented in the UI" instance = "Details about the instance. Contains the connector configuration field which holds the information. If a database is involved it would be the DB connection details, if it is a file it would be the file path, etc" sink = true extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source" icon = "file.png" description = "A description for the connector" author = "The connector author" } ... ] }
View supported connectors:
lenses-cli connectors --supported
See also source connectors configuration.
External secret providers allow for indirect references to be placed in an application’s configuration so that secrets are not exposed in the Worker API endpoints of Kafka Connect.
Connectors are also supported by the CLI to enable automation scenarios.
CLI - API
Does any Kafka Connector work with Lenses?
Yes! Lenses connects to the cluster and manages any available plugins.
See how to configure supported connectors metadata
Does Lenses.io offers Connectors for Kafka Connect?
Yes, Lenses.io is contributing to a number of Kafka Connectors open-sourced in the Kafka community. see the full list of supported connectors
Can we spin up new connect clusters?
You can connect all your connect clusters to Lenses, via Connections in the UI or automate via CLI/API, at any time through Lenses UI or Lenses CLI.
Can we have dynamically configuration of new connect clusters?
Yes. You can dynamically configure Lenses to connect to your Kafka Connect clusters at any time, through Lenses UI, Lenses CLI, or by using yaml files.
Can we have historic metrics of the connectors?
Not in Lenses per se. You can on the other hand use the Grafana templates that are provided by Lenses.io for historic metrics on your Kafka Connect
Can we get specific resource metrcis for each connector? ie. CPU, Memory etc
Not in Lenses per se. See our grafana dashboards
On this page
