Managing Schemas

Lenses supports AVRO which is a great choice for a data serialization format:

  1. It has a direct mapping to and from JSON.
  2. It has a very compact format. The bulk of JSON, repeating every field name with every single record, is what makes JSON inefficient for high-volume usage.
  3. It is very fast.
  4. It has great bindings for a wide variety of programming languages so you can generate Java objects that make working with event data easier, but it does not require code generation so tools can be written generically for any data stream.
  5. It has a rich, extensible schema language defined in pure JSON.
  6. It has the best notion of compatibility for evolving your data over time.

Lenses allows you to manage schemas and their evolution. Navigating to the Schemas option in the side menu will land you to the schema overview page, which allows you to:

  1. View and search schemas.
  2. Change the global compatibility levels.
  3. Create new schemas.
../../_images/schema.png

Create new schemas

To create a new schema select the New Schema button on the right-hand side. This presents a template AVRO schema in JSON format. Construct your schema and submit the request. Lenses will validate the schema before it is deployed.

../../_images/schema-new.png

Note

Both the key and the value of a Kafka message can have a schema. Add the -value or -key suffix to the subject accordingly.

Edit a schema

To edit a schema, select the schema you wish to edit. This will display schema both in JSON and a tabular format. Click the edit button and apply your changes. Before the edits are applied Lenses will validate that the changes match the schema evolution rules.

../../_images/schema-edit.png

View schema history

Schemas can evolve and Lenses not only allows you to browse the different versions of schemas as they evolve over time but also can show a Git style diff between different schema versions. For the selected schema select the Version button to view the different versions or Actions -> View History for displaying the changes between them.

../../_images/schema-diff.png

Important

You might want to have control over who makes schemas changes. By default, schema creation, deletion and editing is only allowed to users with WRITE access, such as admin and write.