Prerequisites

Permissions 

Permissions required:

• To view an SQL Processor:
  • Application permission: at least View level of the SQL Processors
  • Data Namespaces permissions: at least Show Topic for each input and output topic
• To manage an SQL Processor lifecycle (start, stop, scale and delete):
  • Application permission: at least Manage level of the SQL Processors
  • Data Namespaces permissions: at least Show Topic for each input and output topic
• To create a new SQL Processor:
  • Application permission: at least Manage level of the SQL Processors
  • Data Namespaces permissions: at least Query Topic for each input topic
  • Data Namespaces permissions: at least Insert Data for each output topic

Apart from that, when running SQL processors in CONNECT mode, permissions for given Kafka Connect cluster should be present to view/manage/create a processor running on this cluster.

Schemas must be available for Structured Data 

To support features like:

• Inference of output schemas
• Creation-time validation of input query
• Selections
• Expressions

LSQL Engine Streaming mode needs to have up-to-date schema information for all structured topics that are used as input in a given query. In this context, structured means topics that are using complex storage formats like AVRO or JSON.

INSERT INTO daily-item-purchases-stats
SELECT STREAM
    itemId
    , COUNT(*) as dailyPurchases
    , AVG(price / quantity) as average_per_unit
FROM purchases
WINDOW BY TUMBLE 1d
GROUP BY itemId;

For the above query, for example, the purchases topic will need to have a value set to a structured format and a valid schema will need to already have been configured in Lenses. In such schema, fields itemId, price and quantity must be defined, the latter two being of a numerical type.

These requirements ensure the Engine will always in a position to know what kind of data it will be working with, guaranteeing at the same time that all obvious errors are caught before a query is submitted.