A Kafka Connect source connector to write events from Apache Pulsar to Apache Kafka.
The following KCQL is supported:
INSERT INTO kafka_topic SELECT FIELDS, ... FROM pulsar_topic [WITHSUBSCRIPTION=[SHARED|EXCLUSIVE|FAILOVER]] [WITHCONVERTER=`myclass`]
Selection of fields from the Pulsar message is not supported.
Examples:
-- shared subscription and batching INSERT INTO kafka-topicA SELECT * FROM persistent://lenses/standalone/connect/kafka-topic BATCH = 100 WITHSUBSCRIPTION=SHARED -- default failover INSERT INTO kafka-topicA SELECT * FROM persistent://lenses/standalone/connect/kafka-topic WITHCONVERTER=`myclass`
The connector supports the following Pulsar subscription types set using the WITHSUBSCRIPTION clause:
The connector supports converters to handle different messages payload format in the source topic. See source record converters.
export CONNECTOR=pulsar docker-compose up -d pulsar
Login into the pulsar container:
docker exec -ti pulsar /bin/bash
Run the following command to generate messages:
bin/pulsar-client produce \ persistent://lenses/standalone/connect/kafka-topic \ --messages 'hello-pulsar'
If you are using Lenses, login into Lenses and navigate to the connectors page, select Pulsar as the source and paste the following:
name=pulsar connector.class=com.datamountaineer.streamreactor.connect.pulsar.source.PulsarSourceConnector tasks.max=1 connect.pulsar.kcql=INSERT INTO pulsar SELECT * FROM persistent://lenses/standalone/connect/kafka-topic BATCH = 10 connect.pulsar.hosts=pulsar://pulsar:6650 connect.progress.enabled=true
To start the connector using the command line, log into the lenses-box container:
docker exec -ti lenses-box /bin/bash
and create a connector.properties file containing the properties above.
Create the connector, with the connect-cli:
connect-cli create pulsar < connector.properties
Wait for the connector to start and check it’s running:
connect-cli status pulsar
Check the records in Lenses or with via the console:
kafka-avro-console-consumer \ --bootstrap-server localhost:9092 \ --topic pulsar \ --from-beginning
Bring down the stack:
docker-compose down
On this page
