Kafka Connect

Allow developers to quickly build robust, durable, scalable and secure pipelines in and out of Kafka with Kafka Connect.

See how to configure Lenses and integrate Kafka Connect .

For supported Lenses Connectors see the Stream reactor.

Bring your own connector 

Created your own Connector or want to use another community Connector?

Lenses can manage any connector that implements the Kafka Connector interfaces, ensure your Connector returns its ConfigDef in the config() method of Connector class.

Configure Lenses to recognise metadata about your connector in the lenses.conf file:

connectors.info = [
{
    class.name = "The connector full classpath"
    name = "The name which will be presented in the UI"
    instance = "Details about the instance"
    sink = true
    extractor.class = io.lenses.config.kafka.connect.SimpleTopicsExtractor
    icon = "file.png"
    description = "A description for the connector"
    author = "The connector author"
}]

Source connectors 

Source connectors do not have a standard way to identify target topics in Kafka. The extractor.class option, as seen above, allows Lenses to identify which Kafka topics the connector writes to. When using this extractor it is also expected to provide a property configuration which specifies the field within the connector runtime configuration containing the topics to publish to. Here is an example for a file stream source connector:

class.name = "org.apache.kafka.connect.file.FileStreamSource"
name = "File"
instance = "file"
sink = false
property = "topic"
extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"

See the Configuration settings for more details.