SQL Streaming

Streaming SQL processing of real-time data can run in 2 modes:

SQL in Process 

Streaming SQL applications will be run as part of Lenses’ own process, sharing resources, memory and CPU time with the rest of the platform.

Set the execution configuration to IN_PROC

# Set up Lenses SQL processing engine
lenses.sql.execution.mode = "IN_PROC"

Set the directory to store the internal state of the SQL Processors:

lenses.sql.state.dir = "/tmp/lenses-sql-kstream-state"

Connections management 

Since Lenses 4.0, connection details are treated as secrets and stored in the internal database. Among other reasons, that change allows Lenses to bring governance on the connections, and subsequently, to the systems that Processors connect to.

SQL processors uses the same connection details that Lenses uses to speak to Kafka and Schema Registry. The following properties are mounted, if present, on the file system for each processor:

  • Kafka
    1. SSLTruststore
    2. SSLKeystore
  • Schema Registry
    1. SSL Keystore
    2. SSL Truststore

The files structure created by applications is the following: /run/[lenses_installation_id]/applications/

SQL on Kubernetes 

Kubernetes can be used to deploy SQL Processors to. To configure Kubernetes set the mode to KUBERNETES and configure the location of the kubeconfig file.

When Lenses is deployed inside of Kubernetes the lenses.kubernetes.config.file configuration entry should be set to an empty string. The Kubernetes client will auto configure from the pod it is deployed in.

The streaming SQL docker image live in Dockerhub.

lenses.sql.execution.mode = KUBERNETES
# kubernetes configuration
lenses.kubernetes.config.file = "/home/lenses/.kube/config"
lenses.kubernetes.service.account = "default"
#lenses.kubernetes.processor.image.name = "" # Only needed if you use a custom image
#lenses.kubernetes.processor.image.tag = ""  # Only needed if you use a custom image

# Only needed if you want to tune the buffer size for incoming events from Kubernetes
#lenses.deployments.errors.buffer.size = 1000

# Only needed if you want to tune the buffer size for incoming errors from Kubernetes WS communication
#lenses.deployments.events.buffer.size = 10000

Kerberos 

If Kerberos is required set the JAAS file for the SQL processors. Use the following configuration:

   lenses.kubernetes.processor.kafka.settings.security.protocol = SASL_PLAINTEXT
   lenses.kubernetes.processor.jaas                  = "/jaas-processors.conf"
   lenses.kubernetes.processor.kafka.settings.keytab = "/processor.keytab"
   lenses.kubernetes.processor.krb5                  = "/etc/krb5.conf"

For the jaas.conf, replace the paths so that the keytab points to /mnt/secrets/kafka/keytab as the contents are required to be mounted in the Pods in a well known location.

KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/mnt/secrets/kafka/keytab"
  storeKey=true
  useTicketCache=false
  serviceName="kafka"
  principal="principal@MYREALM";
};

/*
  Optional section for authentication to zookeeper
  Please also remember to set lenses.zookeeper.security.enabled=true
*/
Client {
  com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="/mnt/secrets/kafka/keytab"
   storeKey=true
   useTicketCache=false
   principal="principal@MYREALM";
};

Krb5 files cannot include the includeDir property as this will not resolve in the Pods. If you require this, extend the SQL Processor image and update Lenses to use your custom image.

Custom Serde 

Custom serdes should be embedded in a new Lenses SQL processor Docker image.

To build a custom Docker image, create the following directory structure:

mkdir -p processor-docker/serde

Copy your serde jar files under processor-docker/serde.

Create Dockerfile containing:

FROM lensesioextra/sql-processor:4.2

ADD serde /opt/serde
ENV LENSES_SQL_RUNNERS_SERDE_CLASSPATH_OPTS=/opt/serde

Build the Docker.

cd processor-docker
docker build -t example/lsql-processor .

Once the image is deployed in your registry, please set Lenses to use it (lenses.conf):

lenses.kubernetes.processor.image.name = "your/image-name"
lenses.kubernetes.processor.image.tag = "your-tag"