4.1
Prerequisites
Permissions
Permissions required:
- To view an SQL Processor:
- Application permission: at least
View
level of theSQL Processors
- Data Namespaces permissions: at least
Show Topic
for each input and output topic
- Application permission: at least
- To manage an SQL Processor lifecycle (start, stop, scale and delete):
- Application permission: at least
Manage
level of theSQL Processors
- Data Namespaces permissions: at least
Show Topic
for each input and output topic
- Application permission: at least
- To create a new SQL Processor:
- Application permission: at least
Manage
level of theSQL Processors
- Data Namespaces permissions: at least
Query Topic
for each input topic - Data Namespaces permissions: at least
Insert Data
for each output topic
- Application permission: at least
Apart from that, when running SQL processors in CONNECT mode, permissions for given Kafka Connect cluster should be present to view/manage/create a processor running on this cluster.
Schemas must be available for Structured Data
To support features like:
- Inference of output schemas
- Creation-time validation of input query
- Selections
- Expressions
LSQL Engine Streaming mode needs to have up-to-date schema information for all structured topics that are used as input
in a given query. In this context, structured means topics that are using complex storage formats like AVRO
or JSON
.
INSERT INTO daily-item-purchases-stats
SELECT STREAM
itemId
, COUNT(*) as dailyPurchases
, AVG(price / quantity) as average_per_unit
FROM purchases
WINDOW BY TUMBLE 1d
GROUP BY itemId;
For the above query, for example, the purchases
topic will need to have a value set to a structured format and a
valid schema will need to already have been configured in Lenses. In such schema, fields itemId
, price
and
quantity
must be defined, the latter two being of a numerical type.
These requirements ensure the Engine will always in a position to know what kind of data it will be working with, guaranteeing at the same time that all obvious errors are caught before a query is submitted.
Old generation processors
Starting with Lenses 4.0, there is a new and improved SQL stream processing engine. Processors setup via earlier versions are still being managed by Lenses, after an upgrade. However, there is no support for creating SQL processors using the old engine - its syntax is deprecated; a label will appear in the user interface to highlight those specific entries.