Consumer lag monitoring

Introduction 

Consumer Groups or Consumers are …

Required permission 

Access Management & permissions

Monitor Kafka Consumers 

Lenses continuously monitors Kafka consumers, applications consuming data from your cluster. It uses the native Kafka client to calculate in real-time metrics around the Lag per partition and status information. Consumer Groups represent Kafka applications that consume data from one or more Kafka topics. The consumers typically commit their offsets to a system topic called __consumer_offsets to persist their state (until which point they have consumed).

You can view all the consumers by navigating to Consumers

Inactive Consumers

Inactive consumers are not enabled by default so if you want to access them select the Show Inactive option

Consumer Lag & State 

The Consumer Lag represents the lag between the data in topic ( data produced ) and the data that have been processed by the consumer application. Every Consumer, consumes data at their own pace and when the rate of produced data exceeds the rate which is getting consumed, this creates a lag. So the lag, is the difference between the latest offset in the topic partition and the latest committed offset by the consumer.

The red bar under the consumer lag column, indicates the lag range (minimum and maximum lag per partition). When a single partition is involved the lag is a single number.

To view the exact lag per partition navigate to the consumer details and see how to add consumer lag alerts.

Consumer Instances 

Navigate to each Consumer to view the details for its instances and how the topic partitions have been distributed to each instance as well as the offset details and lag for each topic-partition.

• Coordinator ID is the Kafka broker who acts as the coordinator for this consumer group
• Latest Offset is the latest offset of a partition
• Current Offset is the current offset until which the consumer has read the message

Consumer Groups may consume from multiple topics at the same time and the load is automatically distributed across each application instance in the group. Your applications, Kafka Connectors and Streaming SQL have a consumer group when they are consuming events from Kafka.

Consumer Lag Alerts 

You can add custom threshold alerts for the consumer lag. For each consumer group and topic combination you can set an alert trigger which will occur when the threshold exceeds.

Click on the View Alerts button to see the associated alerts with this consumer group or add a new one.

Consumer Lag for Prometheus 

If you want to track in a time-series system, such as Prometheus / Grafana the historic values of the Consumer Lag, just add Lenses as a target. Lenses calculates and makes available consumer lag info in a prometheus compatible format under the Lenses API:

<your-lenses-host>/metrics

You can then hook this to your Grafana:

Manage Consumer Offsets 

With Lenses you can manage the Consumer offsets. You can replay or skip messages for a partition or go back in time for multiple partitions. To edit the offsets your Consumer must be inactive, so that is not committing offsets while reading the topic. If your Consumers are active and committing their offsets back to Kafka, Lenses has the options for changing offsets disabled.

Find inactive consumers:

Change topic-partition Consumer offset 

To change the offset for the inactive Consumer navigate to the Consumer details. For each topic-partition you can change the offset:

Change multiple Consumer offsets 

From the Actions menu, select Change multiple offsets

And select the one of the options:

• Set Offsets at Start - Sets offsets to the starting offset of each topic to consume the topic from the beginning
• Set Offsets at the End - Sets offsets to the end offset of each topic so consumer skips the messages
• Set Specific Date and Time - Sets the offsets to a point in time ( will be specific to the client’s timezone )

If your Consumer consumes data from multiple topics you can select to apply the change to all or conditionally to specific topics

To replay records for a consumer group, the consumer must be stopped in order not to commit offsets while consuming. Therefore the option in Lenses is disabled when the consumer is up and running.

FAQ 

How can I see my Consumers in Topology

Any registered application can be viewed as part of the overall topology. For applications registered via the JVM we can also view the collected metrics.

How can I integrate Consumer Lag metrics to my monitoring system?

Consumer lag is exposed via the /metrics API

How can I delete a Consumer?

At the moment Lenses is not giving you the ability to delete a Consumer group from its UI or API. You can use the command line tools of Kafka

kafka-consumer-groups.sh --bootstrap-server {Kafka instance connection address} --delete --group {consumer group name}