Release notes

4.2.0 

• All

  • Ensure connector version is retained by connectors
  • Lenses branding ASCII art updates

• AWS S3 Sink Connector

  • Improves the error in case the input on BytesFormatWriter is not binary
  • Support for ByteBuffer which may be presented by Connect as bytes

4.1.0 

• All

  • Scala upgrade to 2.13.10
  • Dependency upgrades
  • Upgrade to Kafka 3.3.0
  • SimpleJsonConverter - Fixes mismatching schema error.

• AWS S3 Sink Connector

  • Add connection pool config
  • Add Short type support
  • Support null values
  • Enabling Compression Codecs for Avro and Parquet
  • Switch to AWS client by default
  • Add option to add a padding when writing files, so that files can be restored in order by the source
  • Enable wildcard syntax to support multiple topics without additional configuration.

• AWS S3 Source Connector

  • Add connection pool config
  • Retain partitions from filename or regex
  • Switch to AWS client by default

• MQTT Source Connector

  • Allow toggling the skipping of MQTT Duplicates

• MQTT Sink Connector

  • Functionality to ensure unique MQTT Client ID is used for MQTT sink

• Elastic6 & Elastic7 Sink Connectors

  • Fixing issue with missing null values

4.0.0 

• All

  • Scala 2.13 Upgrade
  • Gradle to SBT Migration
  • Producing multiple artifacts supporting both Kafka 2.8 and Kafka 3.1.
  • Upgrade to newer dependencies to reduce CVE count
  • Switch e2e tests from Java to Scala.

• AWS S3 Sink Connector

  • Optimal seek algorithm - see documentation for more detail.
  • Parquet data size flushing fixes.
  • Adding date partitioning capability - see documentation for more detail.
  • Adding switch to use official AWS library - see documentation for more detail.
  • Add AWS STS dependency to ensure correct operation when assuming roles with a web identity token.
  • Provide better debugging in case of exceptions.

• FTP Source Connector

  • Fixes to slice mode support.

• Hazelcast Sink Connector

  • Upgrade to HazelCast 4.2.4. The configuration model has changed and now uses clusters instead of username and password configuration.

• Hive Sink Connector

  • Update of parquet functionality to ensure operation with Parquet 1.12.2.
  • Support for Hive 3.1.3.

• JMS Connector

  • Enable protobuf support.

• Pulsar

  • Upgrade to Pulsar 2.10 and associated refactor to support new client API.

3.0.1 

• All

  • Replace Log4j with Logback to overcome CVE-2021-44228
  • Bringing code from legacy dependencies inside of project

• Cassandra Sink Connector

  • Ensuring the table name is logged on encountering an InvalidQueryException

• HBase Sink Connector

  • Alleviate possible race condition

3.0.0 

• All

  • Move to KCQL 2.8.9
  • Change sys.errors to ConnectExceptions
  • Additional testing with TestContainers
  • Licence scan report and status

• AWS S3 Sink Connector

  • S3 Source Offset Fix
  • Fix JSON & Text newline detection when running in certain Docker images
  • Byte handling fixes
  • Partitioning of nested data
  • Error handling and retry logic
  • Handle preCommit with null currentOffsets
  • Remove bucket validation on startup
  • Enabled simpler management of default flush values.
  • Local write mode - build locally, then ship
  • Deprecating old properties, however rewriting them to the new properties to ensure backwards compatibility.
  • Adding the capability to specify properties in yaml configuration
  • Rework exception handling. Refactoring errors to use Either[X,Y] return types where possible instead of throwing exceptions.
  • Ensuring task can be stopped gracefully if it has not been started yet
  • ContextReader testing and refactor
  • Adding a simple state model to the S3Writer to ensure that states and transitions are kept consistent. This can be improved in time.

• AWS S3 Source Connector

  • Change order of match to avoid scala.MatchError
  • S3 Source rewritten to be more efficient and use the natural ordering of S3 keys
  • Region is necessary when using the AWS client

• Cassandra Sink & Source Connectors

  • Add connection and read client timeout

• FTP Connector

  • Support for Secure File Transfer Protocol

• Hive Sink Connector

  • Array Support
  • Kerberos debug flag added

• Influx DB Sink

  • Bump influxdb-java from version 2.9 to 2.29
  • Added array handling support

• MongoDB Sink Connector

  • Nested Fields Support

• Redis Sink Connector

  • Fix Redis Pubsub Writer
  • Add support for json and json with schema

2.1.3 

• Move to connect-common 2.0.5 that adds complex type support to KCQL

2.1.2 

• AWS S3 Sink Connector
  • Prevent null pointer exception in converters when maps are presented will null values
  • Offset reader optimisation to reduce S3 load
  • Ensuring that commit only occurs after the preconfigured time interval when using WITH_FLUSH_INTERVAL
• AWS S3 Source Connector (New Connector)
• Cassandra Source Connector
  • Add Bucket Timeseries Mode
  • Reduction of logging noise
  • Proper handling of uninitialized connections on task stop()
• Elasticsearch Sink Connector
  • Update default port
• Hive Sink
  • Improve Orc format handling
  • Fixing issues with partitioning by non-string keys
• Hive Source
  • Ensuring newly written files can be read by the hive connector by introduction of a refresh frequency configuration option.
• Redis Sink
  • Correct Redis writer initialisation

2.1.0 

• AWS S3 Sink Connector
• Elasticsearch 7 Support

2.0.1 

• Hive Source

  • Rename option connect.hive.hive.metastore to connect.hive.metastore
  • Rename option connect.hive.hive.metastore.uris to connect.hive.metastore.uris

• Fix Elastic start up NPE

• Fix to correct batch size extraction from KCQL on Pulsar

2.0.0 

• Move to Scala 2.12
• Move to Kafka 2.4.1 and Confluent 5.4
• Deprecated:
  • Druid Sink (not scala 2.12 compatible)
  • Elastic Sink (not scala 2.12 compatible)
  • Elastic5 Sink(not scala 2.12 compatible)
• Redis
  • Add support for Redis Streams
• Cassandra
  • Add support for setting the LoadBalancer policy on the Cassandra Sink
• ReThinkDB
  • Use SSL connection on Rethink initialize tables is ssl set
• FTP Source
  • Respect connect.ftp.max.poll.records when reading slices
• MQTT Source
  • Allow lookup of avro schema files with wildcard subscriptions