Lenses integrates with Kafka to give observability and management capabilities over your topics, the data, schemas and metrics. Granular permissions apply for the different feature so that self-service capabilities can be customized to your needs.
If you integrate with Schema Registry (SR), Lenses will read those schemas to enhance the experience. If an SR service isn’t available, Lenses will maintain types and schemas for topics at its own internal store to enable the different features at a field level.
As a user, you can get full control of your topics, query the data, update the configuration, review metrics and add alerts.
The level of permissions in a Kafka topic is fully controlled via Groups Namespaces. From there you can tailor to which topics a group of users has access and what operations they can perform. Review the permission matrix to see all the available operations.
Access Management and permissions
A topic is a named stream of events (or messages). The topic data is partitioned to one or more partitions. The messages on each partition are ordered by a number the offset.
Partition is the unit of scale in Kafka. If a topic has multiple partitions, the clients can produce data in parallel to increase throughput by distributing the partitions across the cluster. The number of partitions also influences the balancing of workload among consumers.
To improve availability, each topic can be replicated onto multiple brokers. For each partition, one of the brokers is the leader, and the other brokers are the followers. Replication works by the followers repeatedly fetching messages from the leader.
The events are key-value pairs with additional metadata can be added in headers. To read/write the data you need to deserialize/serialize in a given format and/or schema.
Lenses has primitive support for the headers and they can be displayed to your results. There is also support for SQL functions to work best with the header data.
The indicators next to the topic name display additional information:
C
compacted
S
system topic
d
deletion
Systems topics (datasets) are used to indicate which topics have been automatically created by the system (ie. __consumer_offsets ). By default they are filtered out, but you can choose to view them by ticking the box on the filter list.
Systems topics (datasets)
__consumer_offsets
The default setting includes Lenses system topics, SQL processors’ Kstreams topics, consumer offsets, schemas, and transactions. You can add topics of your own as well, but it is advised to keep the default ones too, so they will not be transferred to your user topics. The setting takes prefixes so, for example, the lsql_ item matches all topics starting with lsql_.
lsql_
Below is the list of built-in configuration to identify them.
_schemas
_kafka_lenses_
lsql_*
lsql-*
__transaction_state
__topology
__topology__metrics
_confluent*
*-KSTREAM-*
*-TableSource-*
*-changelog
__amazon_msk*
TODO - Move to self service
On this page
