Deploying SQL Processors

Lenses allows you to create SQL processors which are Kafka Streams application written in Lenses SQL. The user experience is similar for each execution mode, however CONNECT and KUBERNETES offer the ability to scale the number of processors up or down and have additional properties such as the cluster and namespace to deploy to.

Create a Lenses SQL Processor

Navigate to the SQL Processors view from the side menu. This view will list and provide details for the currently deployed processors. The execution mode is displayed on the top right alongside the button for creating new processors.

../../_images/sql-processors-main-in-proc.png

Selecting the New Processor option will present you with a modal form allowing you to configure the processor. Each processor requires:

  1. Name - This must be unique.
  2. Runners - The parallelization, how many instances of the Lenses SQL application you want running.
  3. Lenses SQL statement - The SQL code for the stream processing.
  4. Cluster - The cluster to deploy to, either the Connect cluster or Kubernetes cluster. Only applicable for the CONNECT and KUBERNETES execution modes.
  5. Namespace - Kubernetes namespace to deploy to. Only applicable to the KUBERNETES execution mode.
  6. Pipeline - Optional label for Kubernetes which is attached to the pods. Kubernetes will inject this into metrics collected by Prometheus.

The image below shows an example for Kubernetes.

../../_images/sql-processors-create-k8.png

Note

In the KUBERNETES execution mode, it may take a few seconds for Kubernetes to pull and deploy the Lenses SQL docker image and the runners to report in.

Status and Configuration

Processors status and metrics are tracked in real time. Selecting a processor will provide its topology graph generated by the SQL engine. Interacting with the input/output topic(s) nodes on the topology graph drives will end up displaying the data flowing through them.

../../_images/sql-processor-detail.png
../../_images/sql-processors-multi-runners.png

Check the metrics in order to get an understanding of how the Lenses SQL processor is performing. This includes the lag for the source topics.

../../_images/sql-processors-metrics.png

Managing Processors

Lenses allows you to stop, start, scale and delete processors.

Select the desired processor to go to the detailed view and then use the Actions button for the list of available operations.

../../_images/sql-processors-metrics.png

When using the KUBERNETES execution mode, it might take a few seconds for Kubernetes to pull and deploy the Lenses SQL docker image and the runners to report in, or for Kubernetes to terminate and clean up the pods when scaling down.

Any change in a processors configuration is audited and a Last Action message is also displayed.

Important

You might want to control who makes changes to your processor. By default, processor creation, deletion and editing is only allowed for users with WRITE access, such as admin and write.