cloud-common
The source and sink has been the focus of this release.
PROPERTIES('store.envelope'=true)
STOREAS BYTES
PROPERTIES
INSERT INTO ... SELECT ... FROM ... PROPERTIES(property=key, ...)
PARTITIONBY a, `field1.field2`
For installations that have been using the preview version of the S3 connector and are upgrading to the release, there are a few important considerations:
Previously, default padding was enabled for both “offset” and “partition” values starting in June.
However, in version 5.0, the decision to apply default padding to the “offset” value only, leaving the " partition" value without padding. This change was made to enhance compatibility with querying in Athena.
If you have been using a build from the master branch since June, your connectors might have been configured with a different default padding setting.
To maintain consistency and ensure your existing connector configuration remains valid, you will need to use KCQL configuration properties to customize the padding fields accordingly.
INSERT INTO $bucket[:$prefix] SELECT * FROM $topic ... PROPERTIES( 'padding.length.offset'=12, 'padding.length.partition'=12 )
Starting with version 5.0.0, the following configuration keys have been replaced.
In version 4.1, padding options were available but were not enabled by default. At that time, the default padding length, if not specified, was set to 8 characters.
However, starting from version 5.0, padding is now enabled by default, and the default padding length has been increased to 12 characters.
Enabling padding has a notable advantage: it ensures that the files written are fully compatible with the Lenses Stream Reactor S3 Source, enhancing interoperability and data integration.
Sinks created with 4.2.0 and 4.2.1 should retain the padding behaviour, and, therefore should disable padding:
INSERT INTO $bucket[:$prefix] SELECT * FROM $topic ... PROPERTIES ( 'padding.type'=NoOp )
If padding was enabled in 4.1, then the padding length should be specified in the KCQL statement:
INSERT INTO $bucket[:$prefix] SELECT * FROM $topic ... PROPERTIES ( 'padding.length.offset'=12, 'padding.length.partition'=12 )
STOREAS Bytes_***
The Bytes_*** storage format has been removed. If you are using this storage format, you will need to install the 5.0.0-deprecated connector and upgrade the connector instances by changing the class name:
Source Before:
class.name=io.lenses.streamreactor.connect.aws.s3.source.S3SourceConnector ...
Source After:
class.name=io.lenses.streamreactor.connect.aws.s3.source.S3SourceConnectorDeprecated ...
Sink Before:
class.name=io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector ...
Sink After:
class.name=io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnectorDeprecated connect.s3.padding.strategy=NoOp ...
The deprecated connector won’t be developed any further and will be removed in a future release. If you want to talk to us about a migration plan, please get in touch with us at sales@lenses.io.
To migrate to the new configuration, please follow the following steps:
All
AWS S3 Sink Connector
AWS S3 Source Connector
MQTT Source Connector
MQTT Sink Connector
Elastic6 & Elastic7 Sink Connectors
FTP Source Connector
Hazelcast Sink Connector
Hive Sink Connector
JMS Connector
Pulsar
Cassandra Sink Connector
HBase Sink Connector
Cassandra Sink & Source Connectors
FTP Connector
Influx DB Sink
MongoDB Sink Connector
Redis Sink Connector
Hive Source
connect.hive.hive.metastore
connect.hive.metastore
connect.hive.hive.metastore.uris
connect.hive.metastore.uris
Fix Elastic start up NPE
Fix to correct batch size extraction from KCQL on Pulsar
Secret provider
Secret provider release notes
Package Name Changes
Package name changes
On this page
