Release notes

4.1.0

Note: S3 Connector Changes

From Version 4.10, S3 Connectors will use the AWS Client by default. You can revert to the jClouds version by setting the connect.s3.aws.client property.

All
- Scala upgrade to 2.13.10
- Dependency upgrades
- Upgrade to Kafka 3.3.0
- SimpleJsonConverter - Fixes mismatching schema error.
AWS S3 Sink Connector
- Add connection pool config
- Add Short type support
- Support null values
- Enabling Compression Codecs for Avro and Parquet
- Switch to AWS client by default
- Add option to add a padding when writing files, so that files can be restored in order by the source
- Enable wildcard syntax to support multiple topics without additional configuration.
AWS S3 Source Connector
- Add connection pool config
- Retain partitions from filename or regex
- Switch to AWS client by default
MQTT Source Connector
- Allow toggling the skipping of MQTT Duplicates
MQTT Sink Connector
- Functionality to ensure unique MQTT Client ID is used for MQTT sink
Elastic6 & Elastic7 Sink Connectors
- Fixing issue with missing null values

4.0.0

All
- Scala 2.13 Upgrade
- Gradle to SBT Migration
- Producing multiple artifacts supporting both Kafka 2.8 and Kafka 3.1.
- Upgrade to newer dependencies to reduce CVE count
- Switch e2e tests from Java to Scala.
AWS S3 Sink Connector
- Optimal seek algorithm - see documentation for more detail.
- Parquet data size flushing fixes.
- Adding date partitioning capability - see documentation for more detail.
- Adding switch to use official AWS library - see documentation for more detail.
- Add AWS STS dependency to ensure correct operation when assuming roles with a web identity token.
- Provide better debugging in case of exceptions.
FTP Source Connector
- Fixes to slice mode support.
Hazelcast Sink Connector
- Upgrade to HazelCast 4.2.4. The configuration model has changed and now uses clusters instead of username and password configuration.
Hive Sink Connector
- Update of parquet functionality to ensure operation with Parquet 1.12.2.
- Support for Hive 3.1.3.
JMS Connector
- Enable protobuf support.
Pulsar
- Upgrade to Pulsar 2.10 and associated refactor to support new client API.

3.0.1

All
- Replace Log4j with Logback to overcome CVE-2021-44228
- Bringing code from legacy dependencies inside of project
Cassandra Sink Connector
- Ensuring the table name is logged on encountering an InvalidQueryException
HBase Sink Connector
- Alleviate possible race condition

3.0.0

All
- Move to KCQL 2.8.9
- Change sys.errors to ConnectExceptions
- Additional testing with TestContainers
- Licence scan report and status
AWS S3 Sink Connector
- S3 Source Offset Fix
- Fix JSON & Text newline detection when running in certain Docker images
- Byte handling fixes
- Partitioning of nested data
- Error handling and retry logic
- Handle preCommit with null currentOffsets
- Remove bucket validation on startup
- Enabled simpler management of default flush values.
- Local write mode - build locally, then ship
- Deprecating old properties, however rewriting them to the new properties to ensure backwards compatibility.
- Adding the capability to specify properties in yaml configuration
- Rework exception handling. Refactoring errors to use Either[X,Y] return types where possible instead of throwing exceptions.
- Ensuring task can be stopped gracefully if it has not been started yet
- ContextReader testing and refactor
- Adding a simple state model to the S3Writer to ensure that states and transitions are kept consistent. This can be improved in time.
AWS S3 Source Connector
- Change order of match to avoid scala.MatchError
- S3 Source rewritten to be more efficient and use the natural ordering of S3 keys
- Region is necessary when using the AWS client
Cassandra Sink & Source Connectors
- Add connection and read client timeout
FTP Connector
- Support for Secure File Transfer Protocol
Hive Sink Connector
- Array Support
- Kerberos debug flag added
Influx DB Sink
- Bump influxdb-java from version 2.9 to 2.29
- Added array handling support
MongoDB Sink Connector
- Nested Fields Support
Redis Sink Connector
- Fix Redis Pubsub Writer
- Add support for json and json with schema

2.1.3

Move to connect-common 2.0.5 that adds complex type support to KCQL

2.1.2

AWS S3 Sink Connector
- Prevent null pointer exception in converters when maps are presented will null values
- Offset reader optimisation to reduce S3 load
- Ensuring that commit only occurs after the preconfigured time interval when using WITH_FLUSH_INTERVAL
AWS S3 Source Connector (New Connector)
Cassandra Source Connector
- Add Bucket Timeseries Mode
- Reduction of logging noise
- Proper handling of uninitialized connections on task stop()
Elasticsearch Sink Connector
- Update default port
Hive Sink
- Improve Orc format handling
- Fixing issues with partitioning by non-string keys
Hive Source
- Ensuring newly written files can be read by the hive connector by introduction of a refresh frequency configuration option.
Redis Sink
- Correct Redis writer initialisation

2.1.0

AWS S3 Sink Connector
Elasticsearch 7 Support

2.0.1

Hive Source
- Rename option connect.hive.hive.metastore to connect.hive.metastore
- Rename option connect.hive.hive.metastore.uris to connect.hive.metastore.uris
Fix Elastic start up NPE
Fix to correct batch size extraction from KCQL on Pulsar

2.0.0

Move to Scala 2.12
Move to Kafka 2.4.1 and Confluent 5.4
Deprecated:
- Druid Sink (not scala 2.12 compatible)
- Elastic Sink (not scala 2.12 compatible)
- Elastic5 Sink(not scala 2.12 compatible)
Redis
- Add support for Redis Streams
Cassandra
- Add support for setting the LoadBalancer policy on the Cassandra Sink
ReThinkDB
- Use SSL connection on Rethink initialize tables is ssl set
FTP Source
- Respect connect.ftp.max.poll.records when reading slices
MQTT Source
- Allow lookup of avro schema files with wildcard subscriptions