Deployment targets

Lenses SQL processors can be deployed in two different ways:

1. IN_PROC
2. Kubernetes

You can only have one target. To see how to configuration the deployment mode please view the installation section.

In process 

This is the default execution where processors run locally in the same process of Lenses. It’s recommended for development or testing as it’s not scalable and tight to the Lenses deployment (ie. when Lenses restarts so is the processor etc. ). There is a maximum limit of 50 IN_PROC processors.

Kubernetes 

This will deploy and run the processor to a Kubernetes cluster namespace, and can scale with multiple runners. You will only need to create the SQL statements and select your cluster and then Lenses will handle the deployment of the runners and monitor the performance. This is the most recommended way to run processors with Lenses. You will be able to select any configured cluster or namespaces, currently not bound to your permissions group.

Advanced Settings 

With Lenses 5.0 advanced settings can be configured for the Kubernetes pod:

Changing a deployment target 

At the current versions of Lenses you cannot change deployment target from the UI, but you can use Lenses CLI to do some manual steps.

Let’s say for example you want to change a deployment target from “in process” to “kubernetes”, then additional information need to be provided ie. the cluster, the namespace etc.

As a pre-requisite you will have to change Lenses configuration to reflect the desired target.

Step 1: List existing processors

lenses-cli processors list

Step 2: Stop all running processors

lenses-cli processor stop --id <application_id>

Step 3: Export processors to generate the YAML files

lenses-cli export processors

You can also export specific processors:

lenses-cli export processors --id <application_id>

In any case, an apps/sql folder will be created, containing a processor descriptor in YAML format

The YAML file looks like this

name: processor-1
sql: |-
  SET defaults.topic.autocreate=true;
  INSERT INTO sink-topic SELECT STREAM * FROM source-topic;  
runnerCount: 1
cluster: IN-PROC
namespace: Lenses
pipeline: processor-1
appId: lsql-processor-1-1551637290

Step 4: Update the YAML file with Kubernetes details

The updated YAML looks like this:

name: processor-1
sql: |-
  SET defaults.topic.autocreate=true;
  INSERT INTO sink-topic SELECT STREAM * FROM source-topic;  
runnerCount: 1
cluster: my-custom-cluster
namespace: sql-lenses
pipeline: processor-1
appId: lsql-processor-1-1551637290

Step 5: Re-create the processors

lenses-cli processor create ./apps/lenses-sql/*

Step 6: Listing processors again will have the new deployment target

lenses-cli processors list

Step 7: The processors are not started so you can now start them

lenses-cli processor start --id <application_id>