Schema Registry

Working with records serialized in Avro or Protobuf format requires schemas. The simplest way to provide schemas is a Schema Registry.

Supported registries 

Confluent Schema Registry
AWS Glue Schema Registry
Aiven Karapace
Apicurio

Lenses supports 2 types of schema registry:

  1. Confluent Schema Registry.
  2. AWS Glue Schema Registry.

It also supports schema registries with Confluent-compatible API:

  1. Aiven Karapace.
  2. Apicurio. See here for Apicurio’s Confluent-compatible API.

For Karapace and Apicurio, configure them in Lenses as if they were a Confluent schema registry.

Once a Schema Registry connection is defined, Lenses displays all Schema Registry nodes and their statuses (with metrics, if configured so) in Workspace -> Services screen.

Apart from the Connection, there are following static config (lenses.conf) entries that might be used to enable schema deletions.

Copy
## Enable schema deletion in the Lenses UI
## default : false
lenses.schema.registry.delete = true

## When a topic is deleted,
## automatically delete also its associated Schema Registry subjects
## default: false
lenses.schema.registry.cascade.delete = true

Adding a connection 

To add a connection, go to:

Admin / Connections / New Connection / Schema Registry

Select your registry type and enter the details. For AWS Glue, you will link it to a Lenses AWS connection to get the credentials. If you don’t have an AWS connection you can create one.

Modifying a connection 

To edit a connection, go to:

Admin / Connections / [your schema registry connection]

select the Schema registry connection and in actions select Edit.

Connection details 

Like other core services, Schema Registry is managed via Connections. See more about managing connections here.

Multiple connections allowed: NO

Deletion allowed: YES

When managing connections other than the Lenses UI, such information are relevant for Schema Registry connection:

Template name: SchemaRegistry

Constant connection name: schema-registry

Provision yaml examples 

The easiest way to manage Connections is via Lenses UI under their respective pages, however it is also possible to do it directly via API, Helm or Lenses CLI. In such case, some connection type-specific values have to be used. Here are few examples of such configuration in YAML format.

  • Find out more about managing Schema Registry Connection via API
  • Find out more about managing Connections via Lenses CLI provision
  • Find out more about installing Lenses via Helm

Confluent 

Simple configuration, with JMX metrics 

The URLs (nodes) should always have a scheme defined (http:// or https://).

Copy
schema-registry:
  tags: ["tag1"]
  templateName: SchemaRegistry
  configurationObject:
    schemaRegistryUrls:
      - http://my-sr.host1:8081
      - http://my-sr.host2:8081
    ## all metrics properties are optional
    metricsPort: 9581
    metricsType: JMX
    metricsSsl: false

Misc metrics configurations 

Find more about multiple options of configuring services’ metrics (like secured JMX, Jolokia, etc) under Services Metrics

Basic authentication 

For Basic Authentication, define username and password properties.

Copy
schema-registry:
  templateName: SchemaRegistry
  tags: []
  configurationObject:
    schemaRegistryUrls:
      - http://my-sr.host1:8081
      - http://my-sr.host2:8081
    username: my-username
    password: my-password

TLS with custom truststore 

A custom truststore is needed when the Schema Registry is served over TLS (encryption-in-transit) and the Registry’s certificate is not signed by a trusted CA.

Copy
schema-registry:
  templateName: SchemaRegistry
  tags: []
  configurationObject:
    schemaRegistryUrls:
      - https://my-sr.host1:8081
      - https://my-sr.host2:8081
    sslTruststore:
      fileRef:
        filePath: /path/to/my/truststore.jks
    sslTruststorePassword: myPassword

TLS with client authentication 

A custom truststore might be necessary too (see above).

Copy
schema-registry:
  name: schema-registry
  templateName: SchemaRegistry
  tags: []
  configurationObject:
    schemaRegistryUrls:
      - https://my-sr.host1:8081
      - https://my-sr.host2:8081
    sslKeystore:
      fileRef:
        filePath: /path/to/my/keystore.jks
    sslKeyPassword: keyPassword
    sslKeystorePassword: keystorePassword

TLS with Basic Authentication 

A custom truststore might be necessary too (see above).

Copy
schema-registry:
  name: schema-registry
  templateName: SchemaRegistry
  tags: []
  configurationObject:
    schemaRegistryUrls:
      - https://my-sr.host1:8081
      - https://my-sr.host2:8081
    username: test
    password: test

Advanced properties 

In very rare cases some custom properties related to Schema Registry might be necessary. They can be passed as additionalProperties, as shown in below example.

Copy
schema-registry:
  tags: ["tag1"]
  templateName: SchemaRegistry
  configurationObject:
    schemaRegistryUrls:
      - http://my-sr.host1:8081
      - http://my-sr.host2:8081
    additionalProperties:
      specific.avro.reader: "false"
      avro.reflection.allow.null: "false"
      avro.remove.java.properties: "false"
      avro.use.logical.type.converters: "false"

AWS Glue 

Some connections depend on others. One example is the AWS Glue Schema Registry connection, which depends on an AWS connection. This is an example to provision Lenses with an AWS connection named my-aws-connection and an AWS Glue Schema Registry that references it:

Copy
connections:
  schema-registry:
    templateName: AWSGlueSchemaRegistry
    tags: []
    configurationObject:
      accessKeyId:
        reference: my-aws-connection
      secretAccessKey:
        reference: my-aws-connection
      glueRegistryArn:
        value: arn:aws:glue:region:123123123:registry/my-registry
  my-aws-connection:
    templateName: AWS
    tags: []
    configurationObject:
      accessKeyId: ExampleAccessKey
      secretAccessKey: ExampleSecretAccessKey
      region: my-region
--
Last modified: November 18, 2024