Topology

Options Table

Key	Description	Type
class.name	The connector full class name. i.e. org.apache.kafka.connect.file.FileStream	string
name	The name as it will appear in the topology view. For example: File	string
instance	Contains the connector configuration key(-s) the extractor instance will use to get the information. Consider the FTP source the value is set to connect.ftp.monitor.tail,connect.ftp.monitor.update. The extractor will get the result of both those fields and provide a list of involved topics	string
extractor.class	Used only for Connect sources to extract the target topics from the connector runtime configuration.	string
property	The connector configuration key which will contain the output topics. Used only for Connect sources.	string
icon	The path to an icon file the UI will use to display the connector.	string
description	A short sentence to say what the connector does	string
author	Who is providing the connector	string

Configure Lenses to load a custom connector

For a custom connector or a connector for which Lenses does not provide a setting already lenses.connectors.info needs to be set. The teamplate below allows the the connector to work with the topology graph, and also have the RBAC rules applied to it:

  lenses.connectors.info = [
      {
        class.name = "The connector full classpath"
        name = "The name which will be presented in the UI"
        instance = "Details about the instance. Contains the connector configuration field which holds the information. If  a database is involved it would be  the DB connection details, if it is a file it would be the file path, etc"
        sink = true
        extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
        icon = "file.png"
        description = "A description for the connector"
        author = "The connector author"
      }
  ]

To extract the topics information from the connector configuration, source connectors require an extra configuration. The extractor class should be: io.lenses.config.kafka.connect.SimpleTopicsExtractor. Using this extractor requires an extra property configuration. It specifies the field in the connector configuration which determines the topics data is sent to.

Here is an example for the file source:

  lenses.connectors.info = [
    {
      class.name = "org.apache.kafka.connect.file.FileStreamSource"
      name = "File"
      instance = "file"
      sink = false
      property = "topic"
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
    }
  ]

And an example of a splunk sink connector and a debezium sql server connector

  lenses.connectors.info = [
    {
      class.name = "com.splunk.kafka.connect.SplunkSinkConnector"
      name = "Splunk Sink",
      instance = "splunk.hec.uri"
      sink = true,
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
      icon = "splunk.png",
      description = "Stores Kafka data in Splunk"
      docs = "https://github.com/splunk/kafka-connect-splunk",
      author = "Splunk"
    },
    {
      class.name = "io.debezium.connector.sqlserver.SqlServerConnector"
      name = "CDC MySQL"
      instance = "database.hostname"
      sink = false,
      property = "database.history.kafka.topic"
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
      icon = "debezium.png"
      description = "CDC data from RDBMS into Kafka"
      docs = "//debezium.io/docs/connectors/mysql/",
      author = "Debezium"
    }
  ]

Configuration for Docker compose

Follow the configuration below for a Lenses setup via Docker compose. Here is an example for SplunkSinkConnector and debezium sql server connector. The two connectors are already covered, this is used for illustration purpose.

version: '3'
services:
  lenses:
    image: lensesio/lenses:latest
    container_name: lenses
    ...
    environment:
      ...
      LENSES_CONNECTORS_INFO: |
        [
            {
            class.name = "com.splunk.kafka.connect.SplunkSinkConnector"
            name = "Splunk Sink",
            instance = "splunk.hec.uri"
            sink = true,
            extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
            icon = "splunk.png",
            description = "Stores Kafka data in Splunk"
            docs = "https://github.com/splunk/kafka-connect-splunk",
            author = "Splunk"
            },
            {
            class.name = "io.debezium.connector.sqlserver.SqlServerConnector"
            name = "CDC MySQL"
            instance = "database.hostname"
            sink = false,
            property = "database.history.kafka.topic"
            extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
            icon = "debezium.png"
            description = "CDC data from RDBMS into Kafka"
            docs = "//debezium.io/docs/connectors/mysql/",
            author = "Debezium"
            }
        ]
    ...