SQL on Kubernetes

Kubernetes can be used to deploy SQL Processors to. To configure Kubernetes set the mode to KUBERNETES and configure the location of the kubeconfig file. Providing a non-valid or non-existent Kubernetes Configuration file makes Lenses fail fast on startup.

Lenses reloads the Kubernetes configuration file on an interval. This interval can be customized setting lenses.kubernetes.config.reload.interval Lenses configuration property. It is expressed in milliseconds. For further details on all available configuration options for using Kubernetes as a deployment backend for SQL processors, please refer to the Kubernetes options reference documentation entry .

When Lenses is deployed inside of Kubernetes the lenses.kubernetes.config.file configuration entry should be set to an empty string. The Kubernetes client will auto configure from the pod it is deployed in.

To know more about what Permissions Lenses needs on Kubernetes, refer to the Kubernetes Role based access

The streaming SQL docker image lives in Dockerhub.

lenses.sql.execution.mode = KUBERNETES
# kubernetes configuration
lenses.kubernetes.config.file = "/home/lenses/.kube/config"
lenses.kubernetes.service.account = "default"
#lenses.kubernetes.processor.image.name = "" # Only needed if you use a custom image
#lenses.kubernetes.processor.image.tag = ""  # Only needed if you use a custom image

# Only needed if you want to tune the buffer size for incoming events from Kubernetes
#lenses.deployments.errors.buffer.size = 1000

# Only needed if you want to tune the buffer size for incoming errors from Kubernetes WS communication
#lenses.deployments.events.buffer.size = 10000

Kerberos 

SQL processors use the same configuration to access Kafka that Lenses does. It is configured as Kafka connection ( see more ).
When using Kerberos, a Kerberos connection should be used too ( see more ).

krb5.conf files cannot include the includeDir property as this will not resolve in the Pods. If you require this, extend the SQL Processor image and update Lenses to use your custom image.

Custom Serde 

Custom serdes should be embedded in a new Lenses SQL processor Docker image.

To build a custom Docker image, create the following directory structure:

mkdir -p processor-docker/serde

Copy your serde jar files under processor-docker/serde.

Create Dockerfile containing:

FROM lensesioextra/sql-processor:5.0

ADD serde /opt/serde
ENV LENSES_SQL_RUNNERS_SERDE_CLASSPATH_OPTS=/opt/serde

Build the Docker.

cd processor-docker
docker build -t example/lsql-processor .

Once the image is deployed in your registry, please set Lenses to use it (lenses.conf):

lenses.kubernetes.processor.image.name = "your/image-name"
lenses.kubernetes.processor.image.tag = "your-tag"