Kafka topics policies

Introduction

Lenses gives the ability to tailor your topic management. You can enable or restrict users to perform different operations based on granular permissions based on namespaces.

Topic management

There are different supported operations for the users like:

Create topics or allow for topic requests
Manage topic configurations
Add Partitions
Monitor, view metrics and add alerts

Read in Kafka Topics & Schemas

Topic settings

In Lenses, you can allow users to create topics or topic requests. To help users follow your topic strategy, or prevent them from adding values that may be impacting your cluster, you can optionally add Topic Settings for important configurations. Additionally, you can enforce your topic naming strategy by adding rules on the pattern or custom regular expressions.

Required permission

Permission	Type	Description
Kafka Settings / View	Admin	Enables user to view topic settings, ACL’s and quotas
Kafka Settings / Manage	Admin	Enables user to edit topic settings, ACL’s and quotas

How it works

Topic Settings are Lenses settings for Kafka Topics on top of your native Kafka topics configurations. The Topic Settings are empty by default, and they take precedence when Topic Creation is applied via Lenses UI/CLI/API/.

Under certain circumstances, the Topic Settings configured through Lenses might no longer be applicable. For instance, if you set a min replication factor of 3, but the number of active brokers in your Kafka cluster suddenly drops to 2, your settings will enter into an un-applicable state. When this happens, Lenses will lock all API calls around topic creation and configuration until this inconsistency is addressed. This can be done either by bringing the third broker back up, or by decreasing the minimum replication factor to 2.

Topic configuration rules

Parameter	Description	Required	Rules
Min Partitions	Min number of acceptable partition	YES	Must be more or equal to 1
Max Partitions	Max number of acceptable partition	NO	Must be more or equal to 1
Min Replication	Min number of acceptable replication factor	YES	Must be more or equal to 1
Max Replication	Max number of acceptable replication factor	NO	Must be more or equal to 1
Default RetentionTime	Default time a Topic can retain data	NO	Must be positive or -1(Infinite)
Max RetentionTime	Max alowed time a Topic can retain data	YES	Must be positive or -1(Infinite)
Default RetentionSize	Default size a Topic can retain data	NO	Must be positive or -1(Infinite)
Max RetentionSize	Max alowed size a Topic can retain data	YES	Must be positive or -1(Infinite)

Naming rules

Parameter	Description	Required
Naming Pattern	Regex to determine if a Topic name is allowed	NO
Naming Description	Desription for the regex which appears to the Create form for the users	NO

Automate with CLI

The following command can be used to retrieve the Topics Settings details. Output can be either JSON or YAML.

lenses-cli topic-settings --output="JSON"

Use the following command to update the topic settings policies:

lenses-cli topic-settings update \
    --partitions-min 1 --partitions-max 3 \
    --replication-min 1 --replication-max 2 \
    --retention-size-default -1 --retention-size-max -1 \
    --retention-time-default -1 --retention-time-max -1 \
    --naming-pattern="[a-zA-Z]*" --naming-description="Only letters ([a-zA-Z]*)"

Export/Import

To import or export topic settings policies:

lenses-cli export topic-settings --dir="DIRECTORY-PATH"
lenses-cli import topic-settings --dir="DIRECTORY-PATH"

Learn about the CLI

FAQ

Can I specify a custom naming strategy?

Under Topic Name, one can set the value of “Custom” for the characters allowed and provide their own regular expression string and description. The description will be used as a naming hint when creating new topics.

What happens if I have set defaults on the Broker?

Topic settings policies take precedence over Kafka Broker defaults. By default the policies are empty.