Release notes

3.0.1 

• All

  • Replacing log4j logging framework with logback to nullify threat from CVE-2021-44228.
  • Removing legacy dependencies (sql-core, json-sql, kafka-connect-smt) to simplify dependency chain.

• Cassandra Sink Connector

  • Fix for #636 - CassandraJsonWriter does not log the table name for InvalidQueryException

• HBase Sink Connector

  • Fix for #753 - Race condition in StructFieldsExtractorBytes

3.0.0 

• All

  • Move to KCQL 2.8.9
  • Change sys.errors to ConnectExceptions
  • Additional testing with TestContainers
  • Licence scan report and status

• AWS S3 Sink Connector

  • S3 Source Offset Fix
  • Fix JSON & Text newline detection when running in certain Docker images
  • Byte handling fixes
  • Partitioning of nested data
  • Error handling and retry logic
  • Handle preCommit with null currentOffsets
  • Remove bucket validation on startup
  • Enabled simpler management of default flush values.
  • Local write mode - build locally, then ship
  • Deprecating old properties, however rewriting them to the new properties to ensure backwards compatibility.
  • Adding the capability to specify properties in yaml configuration
  • Rework exception handling. Refactoring errors to use Either[X,Y] return types where possible instead of throwing exceptions.
  • Ensuring task can be stopped gracefully if it has not been started yet
  • ContextReader testing and refactor
  • Adding a simple state model to the S3Writer to ensure that states and transitions are kept consistent. This can be improved in time.

• AWS S3 Source Connector

  • Change order of match to avoid scala.MatchError
  • S3 Source rewritten to be more efficient and use the natural ordering of S3 keys
  • Region is necessary when using the AWS client

• Cassandra Sink & Source Connectors

  • Add connection and read client timeout

• FTP Connector

  • Support for Secure File Transfer Protocol

• Hive Sink Connector

  • Array Support
  • Kerberos debug flag added

• Influx DB Sink

  • Bump influxdb-java from version 2.9 to 2.29
  • Added array handling support

• MongoDB Sink Connector

  • Nested Fields Support

• Redis Sink Connector

  • Fix Redis Pubsub Writer
  • Add support for json and json with schema

2.1.3 

Move to connect-common 2.0.5 that adds complex type support to KCQL

2.1.2 

• AWS S3 Sink Connector
  • Prevent null pointer exception in converters when maps are presented will null values
  • Offset reader optimisation to reduce S3 load
  • Ensuring that commit only occurs after the preconfigured time interval when using WITH_FLUSH_INTERVAL
• AWS S3 Source Connector (New Connector)
• Cassandra Source Connector
  • Add Bucket Timeseries Mode
  • Reduction of logging noise
  • Proper handling of uninitialized connections on task stop()
• Elasticsearch Sink Connector
  • Update default port
• Hive Sink
  • Improve Orc format handling
  • Fixing issues with partitioning by non-string keys
• Hive Source
  • Ensuring newly written files can be read by the hive connector by introduction of a refresh frequency configuration option.
• Redis Sink
  • Correct Redis writer initialisation

2.1.0 

• AWS S3 Sink Connector
• Elasticsearch 7 Support

2.0.1 

• Hive Source

  • Rename option connect.hive.hive.metastore to connect.hive.metastore
  • Rename option connect.hive.hive.metastore.uris to connect.hive.metastore.uris

• Fix Elastic start up NPE

• Fix to correct batch size extraction from KCQL on Pulsar

2.0.0 

• Move to Scala 2.12
• Move to Kafka 2.4.1 and Confluent 5.4
• Deprecated:
  • Druid Sink (not scala 2.12 compatible)
  • Elastic Sink (not scala 2.12 compatible)
  • Elastic5 Sink(not scala 2.12 compatible)
• Redis
  • Add support for Redis Streams
• Cassandra
  • Add support for setting the LoadBalancer policy on the Cassandra Sink
• ReThinkDB
  • Use SSL connection on Rethink initialize tables is ssl set
• FTP Source
  • Respect connect.ftp.max.poll.records when reading slices
• MQTT Source
  • Allow lookup of avro schema files with wildcard subscriptions