Dataflow Topology

The Topology is a rich UI that enables the visualization of detailed Data Flow Graphs, ETL Pipelines Stream Processing and Analytics workflows and their relation to native Lenses/Kafka resources, such as Kafka Topics, SQL Processors, and Kafka Connectors.

Topology overview

Lenses Topology is comprised of several different Nodes Types. From Topics, SQL Processors, and Connectors, to External Application. Each Node on the UI is clickable. That way, we can take a close view and look for more information regarding each Node.

Topology Overview Small

Topology screen allows for Search in the Nodes by Node name which helps navigation in bigger application landscapes. We categorized the Nodes, based on their type(i.e: SQL Processors, Connectors etc.) for easier identification.

Topology Search

Kafka Topics 

Kafka Topics, are “stores” on information. In the Topology, they are depicted as folders, and by clicking them we can see more information, about their usages as described in the image below:

  • Information about the Topic (Replication, Partitions, Configs, Schema Format).
  • Information about (Producers, Consumers) and Quick Links to them.
  • Information about (Metrics) such as Rate In/Out and Data Size.
Topology Kafka TopicNode
Topology No KafkaTopic

SQL Processors 

Lenses SQL Processors, allow you to use Streaming SQL to process real-time data. In the Topology, they are depected with a Thunder icon. By clicking them, we can see information such as:

  • Information about the Processor (Status, Runners).
  • Information about (Metrics) such as Rate In/Out and Data Size.
  • Information about the (Inner Topology of the Processor, SQL Statement).
Topology SQL ProcessorNode

Kafka Connectors 

Connectors, are used to either import or export Data to and from your Kafka Cluster. They are represented in Lenses Topology as Puzzle icon. By clicking them, we can see information such as:

  • Information about the Connector (Status, Data Type, Number of Runners).
  • Information about the Connector Taks (Task Status, WorkerId).
  • Information about the Topics used (Name, Quick Link).
Topology Kafka ConnectorNode

External Applications 

Applications deployed outside of Lenses can be registered Lenses’ Topology. They are presented with a Lenses App icon. By clicking them, we can see information such as:

  • Information about the Application Status (Status, Runners).
  • Information about the Application (Owner, Deployment, Tags).
Topology Custom App Node

If you are unfamiliar with our RBAC Security, you can find more on Application Permissions. You can use either the Lenses HTTP API or the JVM Topology Client Library to instrument your custom applications.

Metrics 

Lenses Topology has built-in support for overlaying metrics for each entity. You can now monitor “Input” and “Output” Rates for Records and Data, as well as the “Status” of the Application in question. You can configure the frequency of the metrics received via the configuration

FAQ 

How can I add new nodes to the topology?

All of your processors, connectors and their associated topics are visible in the Topology screen. Any other node must be added by registering the application by following this guide.

Can I see my consumers/producers to the topology?

Any registered application can be viewed as part of the overall topology and see where it fits in the streaming pipeline.

My kafka connector is not automatically showing in the topology. Why?

This might be happening for two reasons:

  • The Connect cluster is not configured in Lenses. You can configure the Connect clusters.
  • The user has not acces to the connected resoureces. A node is visible in the Topology screen only if the user has access to all of its connected resources.

Performance issue in many nodes

When your application landscape becomes too heavy (over 5000 nodes), you might experience performance issues in the topology screen.