Navigate to Apps to view, manage and create new processors.
To create a new processor navigate to Apps, and click the New App => SQL Processor. You will be prompt with the form to fill in.
New App => SQL Processor
When creating a new processor it will not start automatically. You will have to explicitly start the processor in order to start consuming data. When you hit the start button, and depending on your deployment target will deploy the number of runners
ProcessorID is the public unique identifier for an SQL processor. You can customize to any arbitrary string following the bellow rules:
^[a-zA-Z0-9\-\._]+
ProcessorID is used as the Kafka consumer group identifier. That means that you can instruct an SQL processor to build its consumer group and coordinate record ingestion from Kafka between all Processor replicas. Consequently, if the ProcessorId of a given SQL processor is changed, that processor will restart consuming messages from the beginning of the existing records in the topic.
You can set it when creating a new processor:
The ApplicationID is the Lenses unique identifier, is automatically created by Lenses, and cannot be customized. This is unique among all SQL Processors.
Lenses uses the ApplicationID to manage applications. This means that, when Starting, Stopping or Scaling an application, Lenses will use this attribute to pick the right instance.
While it’s an internal id, you can still refer to it via the UI:
You can monitor your processor metrics to ensure that data is processed as expected and all runners are working. You can see specific metrics for data consumption and scale up or down according to your needs or resources capacity.
After creating a processor, the topology view of your SQL will be generated to visualize how the data is processed and what are the steps to produce the output. The nodes are interactive, so you can click and get some more insights on each stage.
Underlying consumer
As the processor consumes data from your topics you can navigate to the underlying Kafka Consumer to set alerts for lag or view more granular information about the consumption. You can use the processorID to detect the processor under the consumers page.
You can also manage the offsets ( replay, skip etc ) from the consumer group, but the processor should not be running. For consumers you will need additional permissions.
To delete a running processor you will need to first Stop it from running. Then the delete option will be available and you will be able to remove the processor.
When deleting a processor deployed in an external target ie. Kubernetes, with one or multiple runners, all runners (ie. relevant pods) will be removed.
If the processor has created topics or schemas as part of its execution, they are not going to be deleted. You can use the processorID to search for relevant topics and schemas as they will be prefixed.
A SQL Processor is a continuously running process and can be deployed as an application based on the deployment target configured to Lenses. If you haven’t configured a deployment target you can still run SQL Processors “in process” but you will be limited in scaling the app and the number of processors you can create.
Scaling means that the application will run with multiple runners to share the load, be resilient and fault tolerant. The maximum number of runners would be the maximum number of partitions your processor consumes.
To scale the runners, you will need to have Lenses configured with a deployment target ie. Kubernetes. Then from the UI you can select the number fo runners to scale up or down:
New runners will start, or existing will stop according to the desired state.
You can click on each runner to get specific performance metrics about consumption, production, errors etc:
Since version 4.0 a new generation of the SQL streaming engine is available in order to create SQL Processors. The new version brought a number of new capabilities, it’s more robust and flexible. The new versions of Lenses fully support the new engine but it is not compatible with older versions of SQL Processors. That means that if you have SQL processors running on version < 4.0.x you can still have some basic capabilities from Lenses interface, but it’s recommended that you migrate to the new SQL streaming.
All the processors that have been created with Lenses 3.2 and older will be marked as “Deprecated” so you can identify and plan to migrate.
If you have processors running in on of the external targets they will automatically be imported:
KUBERNETES
The processors can still be manged via Lenses interface (Start, Stop, Scale, Delete), but the topology graph will not be plotted with the detailed nodes:
The “Processor Graph” of the Deprecated Processor will now be represented by only one(1) node which is called “PROCESSOR”
On this page
