Managing Topics

With Lenses, you can create, delete and change the configuration of topics by navigating to the Topics view on the side menu.

This view displays:

1. A chart of the data flowing across the Kafka cluster being monitored.
2. A topic list and details for each topic.
3. Functionality to create and delete topics.

Create a topic

To create a topic select the New Topic button and enter the required details to create the topic. The minimum requirements are:

1. name,
2. partitions
3. replication.

Additionally, you can supply extra configuration to override the broker default values by selecting the Add Config link. This also allows setting common overrides such as retention time, retention size and clean up policy.

Delete a topic

To delete a topic, select the trash can button for the topic and confirm that you wish Lenses to remove that topic.

Note

A topic will be marked for deletion and Kafka will remove it once the consumers have stopped.

Modify a topic

To alter the configuration of an existing topic, select the topic and then the Config tab.

Important

You might want to have control over who makes topic changes. By default topic creation, deletion and editing is only allowed to users with WRITE access, such as admin and write.

View topic partitions

Detailed information and metrics about a topic can be seen in the topic Partitions tab. It displays the distribution of messages across all the topic partitions, provides detailed information for each partition, but also monitors and alerts on in-sync replicas.

Consumers

Lenses tracks consumers in real time; to view the consumer groups for a given topic select the Consumers tab once you have navigated to the desired topic. This will display:

1. Consumer groups active on this topic
2. The clients in the consumer group
3. The lead broker
4. The consumer lag across the group updated in real time
5. The stability of the consumer group

Viewing live data

Lenses provides the ability to visualize live data in topics by utilizing the Lenses SQL engine. Simply select the topic and you are dropped into the Live Stream tab. This view provides:

1. A SQL window to select which data you want to see from the topic.
2. Ability to inspect the data.
3. Pause and restart the stream.

The SQL window allows you to select and filter fields from both the key and value of the records. For a full description of the capabilities please refer to the Lenses SQL section.

Hovering over the records will automatically pause the stream to allow you to inspect records.

Viewing historical data

Lenses provides the ability to visualize and download historical data in topics by utilizing the Lenses SQL engine. Simply select the topic and select the Data tab. This view provides:

1. A SQL window to select which data you want to see from the topic.
2. An offset slider to scroll through data.
3. A tree view of the data.
4. A grid-like view of the data.
5. A raw JSON view of the data.

The SQL window allows you to select and filter fields from both the key and value of the records. For a full description of the capabilities please refer to the Lenses SQL section.

Inserting new data

Lenses allows you to insert new data into a topic, provided your account has either write or admin permissions to that topic.

You can insert new AVRO or JSON payload by dropping new records onto the UI. Additionally, you can even sample or auto generate messages.

This is usually helpful for your QA team when testing micro-services architectures or for pushing new scheduling messages.

You can insert null as the value on an AVRO topic (i.e. when using compacted topics).

Deleting data

With Lenses you can to delete data from a partition of a topic up to a specific offset.

The action will be audited in the audit-log, the physical data will be removed, and notifications will be pushed via the SSE & Alert APIs for the particular action.

How Lenses identifies data types

The format types of the key and value components are configurable in Lenses on a per table basis. Whenever a new table is added, or Lenses runs for the first time, Lenses will try to detect the storage format for both components. A user can always override the types by setting the values for each topic as seen in this screen-shot:

Important

Only a user with TableStorageWrite permission can set up the table/topic storage format information in Lenses.

Advanced users can instruct the SQL engine to use the values provided in the query code itself. For example, imagine a topic where the message key is an INT and the message value is a JSON, but the user decides the value part should be read as STRING. To achieve that, the following code can be used:

SELECT *
FROM payments

If Lenses fails to read using the storage format specified, it will fall back to the default BYTES storage format.