Messages in Kafka are bytes but they can have a schema, especially if a Schema registry is used. Lenses support schema registries and infers data types and schemas.
In the case of AVRO types Lenses pulls the schema from Schema Registry. It will periodically check for new versions and update its internal index.The naming policy is based on the Schema Registry’s TopicNameStrategy which names the schemas with the topic name plus the -key and -value prefix. Schema References are also supported.
TopicNameStrategy
-key
-value
Alternative strategies, ie. you have multiple types of messages in the same topic with different schemas or types other than AVRO are not currently supported via this integration.
Schemas are used to give a structure to your data and data operations. Lenses uses the schemas to better serve the data, create SQL queries or processors, apply field level data policies, catalog searches and more. In your Kafka ecosystem, it’s recommended to use a Schema Registry for better governance and schema management across apps. Lenses supports this integration for AVRO topics and provides the Schema Registry capabilities.
When Schema Registry isn’t available, apart from the type, Lenses will attempt to auto-detect the schema for this type. The schema will be maintained internally, and also allow the users to amend or correct it. Those schemas in Lenses are represented with an AVRO syntax. Primitive types do not require a schema as they are self-descriptive. For headers schema declaration is not currently supported.
Example of a JSON value:
For every Kafka topic, Lenses tracks and maintains a type for the key and the value - and, in some cases, a Header - with an associated schema where required in order to read, serve, process the data or enable field level capabilities such as catalog search, data protection policies etc.
key
value
Header
When Schema Registry is integrated and there are AVRO topics, Lenses will identify those topics and use those schemas (see Schema Registry integration).
When Schema Registry isn’t available, Lenses will try to auto-detect the type for the key and the value and also their schema. If the detection isn’t successful to a matching type then will default the type to BYTES. You are able to change the type as well as the schema detected by Lenses to better serve the users viewing or processing this topic.
BYTES
JSON, AVRO, PROTOBUF, XML, CSV, STRING, INT,LONG, BYTES (default), as well as custom pluggable formats.
For the AVRO and the PROTOBUF formats, Lenses relies on the configured schema registry to detect the topic format and schema. For other formats, detection happens by inspecting the raw bytes of the first message in the first topic partition.
For semi-structured formats such as JSON and XML, Lenses tries to infer a schema if the message value is an object with one or more fields. Please note that such schema is inferred and set only once: it’s down to you managing its evolution safely (e.g. re-setting the schema when a new field is introduced, avoiding backward incompatible changes, etc).
Custom serdes
Lenses supports custom serializers to support custom data formats.
If custom serdes are configured for your environment you can use it directly from the UI and associate with your topic:
For more on custom formats see
On this page
