Lenses CLI¶

Lenses offers a powerful CLI (command-line tool) built in Go that utilizes the REST and WebSocket APIs of Lenses, to communicate with Apache Kafka and exposes a straightforward way to perform common data engineering and site reliability engineering tasks, such as:

Automate your CI/CD (continuous-integration and continuous-delivery)
Create topics/acls/quotas/schemas
Deploy and manage SQL processors in all modes, including deployment to Kubernetes
Deploy and manage Connectors
Run SQL queries on Apache Kafka both browsing and continuous live queries
Change or retrieve configurations to store in GitHub

Installation¶

Binaries are available for download:

Linux MacOS Windows

Or you can use golangs go get.

$ go get -u github.com/lensesio/lenses-go/cmd/lenses-cli

Getting started¶

The lenses-cli application has a very straightforward way to setup an authenticated user for Lenses communication.

The configure command will guide you through.

$ lenses-cli configure

The result will be saved to the current user’s home directory/.lenses/lenses-cli.yml, it looks like this:

Now you’re ready to move on with firing some commands like lenses-cli processors to list all available SQL processors.

You can still pass the credentials and host via the command flags. They take precedence over the config file. Alternatively you can supply a different configuration file with the --config flag:

lenses-cli --config config.yaml processors

The following precedence is used:

Individual flags, host, user and pass
Config file flag
Config file in $HOME/.lenses/

Tip

If you have an existing file use configure --reset to reset

Passing credentials through flags¶

You can always individually configure and log in on the fly for a single command. This is very helpful when you use multi accounts.

$ lenses-cli --host="http://lenseshost:9991" --user="user" --pass="password" $command

Getting help¶

Each command has a --help flag, this will show you the available flags and a simple input example for the specific command.

For example, if we do lenses-cli --help, we’ll see the available (sub)commands and the global flags of the entire lenses-cli application:

lenses-cli - manage Lenses resources and developer workflow

Usage:
  lenses-cli [command]

Examples:
lenses-cli sql --offsets --stats=2s --stats-end "SELECT * FROM reddit_posts LIMIT 50"

Available Commands:
  acl         Work with an Apache Kafka Access Control Lists
  acls        Print the list of the available Apache Kafka Access Control Lists
  alert       Work with alerts
  alerts      List and work with the registered alerts
  configs     Print the whole lenses box configs
  configure   Setup your environment for extensive CLI use. Create and save the required CLI configuration and client credentials
  connector   Get information about a particular connector based on its name
  connectors  List of active connectors' names
  docs        Generate markdown commands documentation on cwd or any specific directory
  help        Help about any command
  live        Live sql provides "real-time" sql queries with your lenses box
  logout      Revoke the access token
  mode        Print the configuration's exeuction mode
  processor   Work with a particular processor based on the processor id, pause, resume, update runners, delete a processor or create an entirely new processor
  processors  List of all available processors
  quota       Work with particular a quota, create a new quota or update and delete an existing one
  quotas      List of all available quotas
  schema      Work with a particular schema based on its name, get a schema based on the ID or register a new one
  schemas     List all available schemas
  sql         Execute or Validate Only Lenses query (LSQL) on the fly
  topic       Work with a particular topic based on the topic name, retrieve it or create a new one
  topics      List all available topic names
  user        Print some information about the authenticated logged user such as the given roles given by the lenses administrator

Flags:
      --config string    load or save the host, user, pass and debug fields from or to a configuration file (yaml, toml or json)
      --debug            print some information that are necessary for debugging
  -h, --help             help for lenses-cli
      --host string      --host=https://example.com
      --pass string      --pass=MyPassword
      --timeout string   --timeout=1h The timeout for connection establishment
      --token string     --token=DSAUH321S%423#32$321ZXN
      --user string      --user=MyUser
      --version          print the current version of the lenses-cli

Use "lenses-cli [command] --help" for more information about a command.

Another example is lenses-cli processors --help, which outputs something like that:

List of all available processors

Usage:
  lenses-cli processors [flags]

Examples:
lenses-cli processors

Flags:
      --clusterName string   select by cluster name, available only in CONNECT and KUBERNETES mode
  -h, --help                 help for processors
      --name string          select by processor name, available only in CONNECT and KUBERNETES mode
      --namespace string     select by namespace, available only in KUBERNETES mode
      --no-pretty            if declared then the result will be not be prettified for humans
  -q, --query string         the jmespath query to further filter results

Tip

Use the -h flag on each subcommand to see the available options.

Global Flags¶

The cli has a number of global flags for all commands:

`--config`	Use the Lenses host and credentials from a file rather than the default in `$HOME/.lenses/`.
`--debug`	Print debug information.
`--pass`	User password
`--user`	User username
`--timeout`	Command timeout.
`--token`	Service account token.
`--host`	The Lenses host to connect to .
`--version`	The version of the CLI.

Common Flags¶

Common flags are not global flags but they can be accessed by the majority of the available commands. For example, the --no-pretty and --query flags can be passed through all commands that fetch and return a JSON-formatted result.

`--no-pretty`	disable the pretty format for JSON output of commands (default false).
`--query`	a jmespath query expression. This allows for querying the JSON output of commands. For more details view the JMESPATH documentation.
`--no-newline`	remove line breakers between string results output of commands (default false).

Access Control Lists¶

Allows view and create, update and delete operations on Apache Kafka topic ACLs.

View ACLs

View all ACLs

$ lenses-cli acls

[
 {
    "resourceType": "Topic",
    "resourceName": "transactions",
    "principal": "principalType:principalName",
    "permissionType": "Allow",
    "host": "*",
    "operation": "Read"
 }
]

Set/Update ACLs

Modify and create ACLs

$ lenses-cli acl set \
    --resourceType="Topic" \
    --resourceName="transactions" \
    --principal="principalType:principalName" \
    --permissionType="Allow" \
    --acl-host="*" \
    --operation="Read"``

ACLs can also be created from a file:

$ lenses-cli acl create ./file.yaml

Delete ACLs

Delete ACLs

$ lenses-cli acl delete \
    --resourceType="Topic" \
    --resourceName="transactions" \
    --principal="principalType:principalName" \
    --permissionType="Allow" \
    --acl-host="*" \
    --operation="Read"``

ACLs can also be deleted from a file:

$ lenses-cli acl delete ./file.yaml

file.yaml:

ResourceType: Topic
ResourceName: transactions
Principal: principalType:principalName
PermissionType: Allow
Host: "*"
Operation: Read

Note

Command line flags are ignored when loading from files

Lenses Configs¶

This command allows you to view the main configurations that Lenses exposes.

$ lenses-cli configs

Example output:

{
...snip

"lenses.ip": "0.0.0.0",
"lenses.jmx.port": 24027,
"lenses.kafka.brokers": "SASL_PLAINTEXT://kafka1:19094,SASL_PLAINTEXT://kafka2:19094",
"lenses.kubernetes.config.file": "",
"lenses.kubernetes.image.name": "eu.gcr.io/k8-engine/lenses-sql-processor-jenkins",
"lenses.kubernetes.image.tag": "v2.0.0",
"lenses.kubernetes.service.account": "default",
"lenses.license.file": "license.json",
"lenses.port": 24017,
"lenses.root.path": "",
"lenses.security.mode": "BASIC",
"lenses.sql.execution.mode": "KUBERNETES",

...snip
}

To see the SQL processors deployment mode you can inspect the lenses.sql.execution.mode. For example, in CONNECT mode a cluster name is required and for KUBERNETES an additional namespace is required to deploy processors.

$ lenses-cli mode
IN_PROC

$ lenses-cli configs mode
IN_PROC

Get single entry config

Returns an individual config setting.

$ lenses-cli configs lenses.port
24017

Connectors¶

Create, modify, pause, restart and remove Apache Kafka Connect connectors.

View Connectors

List currently deployed connectors, optionally filter by name, cluster and namespace:

# list connectors in the cluster dev
$ lenses-cli connectors --clusterName="dev"

Plugins

View the available plugins per cluster or filter by cluster:

lenses-cli connectors plugins --clusterName="dev"

Create

Create connectors via the command line flags or file:

$ lenses-cli connector create --clusterName="dev" --name="connectorName" --config="{\"key\": \"value\"}"

From a file:

$ lenses-cli connector create ./connector.yaml

connector.yaml:

ClusterName: development
Config:
  connector.class: "org.apache.kafka.connect.file.FileStreamSinkConnector"
  file: "/dev/null"
  tasks.max: "1"
  topics: "reddit_posts"
  name: "nullsink"

Note

Command line flags are ignored when loading from files

Configs

View the configuration of a connector:

$ lenses-cli connector config --clusterName="dev" --name="connectorName"

Status

View the status of a connector:

$ lenses-cli connector status --clusterName="dev" --name="connectorName"

Pause

Pause a connector:

$ lenses-cli connector pause --clusterName="dev" --name="connectorName"

Resume

Resume a connector:

$ lenses-cli connector resume --clusterName="dev" --name="connectorName"

Restart

Restart a connector:

$ lenses-cli connector restart --clusterName="dev" --name="connectorName"

Restart a task

Restart a connector task:

$ lenses-cli connector task restart --task=0 --clusterName="dev" --name="connectorName"

View tasks

View tasks for a connector:

$ lenses-cli connector tasks --clusterName="dev" --name="connectorName"``

Task status

View the status of a task for a connector:

$ lenses-cli connector task status --clusterName="dev" --name="connectorName" --task=1

Delete

Delete a connector:

$ lenses-cli connector delete --clusterName="dev" --name="connectorName"

Lenses SQL¶

The CLI allows you to browse and subscribe to topic data with SQL. Please refer to the Lenses SQL documentation for more details.

Browsing

Browse data via the sql command, optionally validate and output stats:

# valid
$ lenses-cli sql --validate "SELECT score, author FROM `reddit_posts` WHERE _vtype='AVRO' AND _ktype='AVRO' LIMIT 50"

# execute with stats every 2 seconds, show offsets at end
$ lenses-cli sql --offsets --stats=5s "SELECT score, author FROM `reddit_posts` WHERE _vtype='AVRO' AND _ktype='AVRO' LIMIT 50"

Live

Live continuous queries update according to the query and never stop until terminated:

$ lenses-cli live sql "SELECT score, author FROM `reddit_posts` WHERE _vtype='AVRO' AND _ktype='AVRO' AND _sample=2 AND _sampleWindow=200"

Queries from files

Both browsing and live queries can be executed from queries stored in files:

# browsing
$ lenses-cli sql ./query.sql

# live continuous
$ lenses-cli live sql ./query.sql

query.sql

SELECT
    score
    , author
FROM `reddit_posts`
WHERE _vtype='AVRO'
AND _ktype='AVRO'
AND _sample=2
AND _sampleWindow=200

Processors¶

The CLI can deploy and manage SQL processors deploy IN_PROC in Lenses or into Apache Kafka Connect or Kubernetes via Lenses.

View processors

View all processors registered, optionally filter by name, cluster and namespace.

$ lenses-cli processors
# filter by namespace
$ lenses-cli processors --namespace lenses-sql

# use the query flag to add further querying
$ lenses-cli processors --query="[?ClusterName == 'IN_PROC'].Name | sort(@) | {Processor_Names_IN_PROC: join(', ', @)}"``

Create processors

Create processors from the command line flags or from a file. For IN_PROC mode you only need the name, for CONNECT mode the name and cluster, for KUBERNETES mode all three:

The following flags are valid for all processor actions

–id (required if –name) is missing
–name (required if –id is missing)
–clusterName (should be passed if KUBERNETES mode)
–namespace (should be passed if KUBERNETES mode)

$ lenses-cli processor create \
    --name="processorName" \
    --runners=1 \
    --clusterName="clusterName" \
    --namespace="ns" \
    --pipeline="prod" \
    --sql="SET autocreate=true;INSERT INTO topicB SELECT * FROM topic1 WHERE  _ktype='BYTES' AND _vtype='AVRO'"

From a file:

$ lenses-cli processor create ./processor.yaml

processor.yaml:

Name: processorName
Runners: 1
SQL: >
  SET autocreate=true;INSERT INTO topicB SELECT * FROM topic1
  WHERE  _ktype='BYTES' AND _vtype='AVRO'

Note

Command line flags are ignored when loading from files

Pause a processor

Pause a running processor. For IN_PROC mode you only need the name, for CONNECT mode the name and cluster, for KUBERNETES mode all three:

$ lenses-cli processor pause --clusterName="clusterName?" --namespace="namespace?" --name="processorName"

Resume a processor

Resume a paused processor. For IN_PROC mode you only need the name, for CONNECT mode the name and cluster, for KUBERNETES mode all three:

$ lenses-cli processor resume --clusterName="clusterName?" --namespace="namespace?" --name="processorName"

Update a processor

Update a processor from the command line flags or file. For IN_PROC mode you only need the name, for CONNECT mode the name and cluster, for KUBERNETES mode all three:

$ lenses-cli processor update \
    --clusterName="clusterName?" \
    --namespace="namespace" \
    --name="processorName" \
    --runners=2`

From a file:

$ lenses-cli processor update ./processor.yaml

processor.yaml:

Name: processorName
Runners: 1
SQL: >
  SET autocreate=true;INSERT INTO topicB SELECT * FROM topic1
  WHERE  _ktype='BYTES' AND _vtype='AVRO'

Warning

You can not update the SQL, only the runners to scale up or down

Delete a processor

Delete a processor by name, cluster and namespace. For IN_PROC mode you only need the name, for CONNECT mode the name and cluster, for KUBERNETES mode all three.

$ lenses-cli processor delete \
    --namespace="namespace" \
    --clusterName="clusterName" \
    --name="processorName

Schemas¶

View, modify and delete schemas and adjust compatibility levels.

View all schemas

View the current schemas registered:

$ lenses-cli schemas

View a schema

Schemas can be filtered by name:

$ lenses-cli schema --name="sea_vessel_position_reports-value"

View schema versions

Schemas versions can be viewed filtered by name:

$ lenses-cli schema versions --name="sea_vessel_position_reports-value"
# view by version
$ lenses-cli schema --name="sea_vessel_position_reports-value" --version="latest"

View schema compatibility levels

Schema compatibility levels can be viewed and filtered by name:

$ lenses-cli schema --name="register1" compatibility

Delete schemas

Delete all versions of a schema by name:

$ lenses-cli schema delete --name="coyote_test_02"

Delete schemas by version

Delete a specific version of a schema:

# (defaults to "latest" if not passed)
$ lenses-cli schema delete-version --name="register1" --version=2

Change compatibility of a Schema

Change the compatibility level of a schema:

$ lenses-cli schema --name="register1" compatibility set FULL

Register a schema

Register a new schema for a name/subject by the command line or file:

$ lenses-cli schema register \
    --name="schemaName" \
    --avro="{ \"type\": \"string\" }"

Register by file:

$ lenses-cli schema register ./schema.yaml

schema.yaml:

Name: schemaName
AvroSchema: |-
  {
    "type": "record",
    "name": "evolution",
    "namespace": "com.landoop",
    "doc": "This is a sample Avro schema to get you started. Please edxt",
    "fields": [
      {
       "name": "name",
       "type": "string"
      },
      {
       "name": "number1",
       "type": "int"
      },
      {
       "name": "number2",
       "type": "float"
      }
    ]
   }

Topics¶

View, create and modify topics:

View topics

View all topics.

$ lenses-cli topics --names --no-json

View a topic

View a specific topic:

$ lenses-cli topic --name=reddit_posts

Update configs

Update a topic configuration:

$ lenses-cli topic update \
    --name="topic2" \
    --configs="[{\"key\": \"max.message.bytes\", \"value\": \"1000020\"}]"

Update from a file:

$ lenses-cli topic update ./topic.yaml

topic.yml:

Name: topicName
Replication: 1
Partitions: 1
Configs:
  max.message.bytes: "1000010"

Note

Command line flags are ignored when loading from files

Create topics

Create topics from the command line or files. If no replication, partitions or configuration is specific the broker defaults are used:

$ lenses-cli topic create \
    --name="topicName" \
    --replication=1 \
    --partitions=1 \
    --configs="{\"max.message.bytes\": \"1000010\"}"

For a file:

$ lenses-cli topic create ./topic.yml

topic.yml:

Name: topicName
Replication: 1
Partitions: 1
Configs:
  max.message.bytes: "1000010"

Delete topics

Delete topics by name:

$ lenses-cli topic delete --name="topic2"

Quotas¶

View, create, modify and delete quotas.

View quotas

View all quotas:

$ lenses-cli quotas

Set quotas for users

Create and update quotas for users from command line flags or files:

$ lenses-cli quota users set [--quota-user="user"] [--quota-client=""] \
    --quota-config="{\"producer_byte_rate\": \"100000\",\"consumer_byte_rate\": \"200000\",\"request_percentage\": \"75\"}"

From a file:

$ lenses-cli quota users set ./quota.yaml

quota.yaml:

# User name for a specific user quota.
# If empty or omitted then create/update the default user quota.
User: user
# All clients, it could be empty as well.
# Change it for a specific client.
ClientID: "*"
Config:
  ProducerByteRate: "100000"
  ConsumerByteRate: "200000"
  RequestPercentage: "75"

Set quotas for clients

Create and update quotas for clients from command line flags or files:

$ lenses-cli quota clients set [--quota-client=""] \
    --quota-config="{\"producer_byte_rate\": \"100000\",\"consumer_byte_rate\": \"200000\",\"request_percentage\": \"75\"}"

From a file:

$ lenses-cli quota clients set ./quota-clients.yaml

quota-clients.yaml:

# Default client quota, it could be empty or "all" as well.
# Change it for a specific client quota.
ClientID: "*"
Config:
  ProducerByteRate: "100000"
  ConsumerByteRate: "200000"
  RequestPercentage: "75"

Delete for users

Delete quotas for users:

# --quota-user, optional. If empty then default user quota, otherwise specific user quota.
# --quota-client, optional. If empty or * then all clients, otherwise specific client.
# arguments, like producer_byte_rate, are not required.
#   Remove user quota config's specific properties, if empty then all properties will be passed on automatically
#   and the user quota will be removed entirely.
$ lenses-cli quota users delete [--quota-client=""] [--quota-user=""] \
    producer_byte_rate consumer_byte_rate request_percentage

# delete for the default user quota
$ lenses-cli quota users delete

# delete for a specific user quota
$ lenses-cli quota users delete \
    --quota-user="user"

# delete for a specific user and client
$ lenses-cli quota users delete \
    --quota-user="user" \
    --quota-client="clientID"

# delete for a specific user quota's property
$ lenses-cli quota users delete \
    --quota-user="user" \
    request_percentage

Delete for clients

Delete quotas for clients:

# --quota-client, optional. If empty then default client quota, otherwise specific client quota.
# arguments, like producer_byte_rate, are not required.
#   Remove client quota config's specific properties, if empty then all properties will be passed on automatically
#   and the client quota will be removed entirely.
$ lenses-cli quota clients delete [--quota-client=""] \
    producer_byte_rate consumer_byte_rate request_percentage

# delete for the default client quota
$ lenses-cli quota clients delete

# delete for a specific client quota
$ lenses-cli quota clients delete \
    --quota-client="clientID"

# delete for a specific client quota's property
$ lenses-cli quota clients delete \
    --quota-client="clientID" \
    request_percentage

Alerts¶

View alerts and modify or delete alert settings.

View all available Alerts

$ lenses-cli alerts [--live]

Note

The --live flag is optional, it displays real-time alert notifications

View all Alert Settings

$ lenses-cli alert settings

View an Alert Setting

$ lenses-cli alert setting --id=2000

{
    "id": 2000,
    "description": "Consumer Lag exceeded",
    "category": "Consumers",
    "enabled": true,
    "docs": "Raises an alert when the consumer lag exceeds the threshold",
    "conditionTemplate": "lag >= $Threshold-Number on group $Consumer-Group and topic $Topic-Name",
    "conditionRegex": "lag >= ([1-9][0-9]*) on group (\\b\\S+\\b) and topic (\\b\\S+\\b)",
    "conditions": {
        "28bbad2b-69bb-4c01-8e37-28e2e7083aa9": "lag >= 100000 on group group and topic topicA",
        "4318a5a7-32e4-43af-9f3f-438e64d14a11": "lag >= 1000000 on group consumerA and topic topicA",
        "92dd89a1-83c0-4251-8610-36fce780a824": "lag >= 1000000 on group minikube.default.london-keyword-count and topic reddit_posts",
        "bdb01792-b79f-44d0-a39e-7f3e0906a89b": "lag >= 100000 on group connect-nullsink and topic topicA",
        "f4dd550f-bcb6-4e02-b28b-18bbd426aaf2": "lag >= 100000 on group groupA and topic topicA"
    },
    "isAvailable": true
}

Enable an Alert Setting

$ lenses-cli alert setting --id=1000 --enable

View all conditions that belong to a specific Alert Setting

$ lenses-cli alert setting conditions --alert=2000

{
    "28bbad2b-69bb-4c01-8e37-28e2e7083aa9": "lag >= 100000 on group group and topic topicA",
    "4318a5a7-32e4-43af-9f3f-438e64d14a11": "lag >= 1000000 on group consumerA and topic topicA",
    "7643d1ec-73f9-4913-95d1-87e62f1ec172": "lag >= 200000 on group groupA and topic topicA",
    "92dd89a1-83c0-4251-8610-36fce780a824": "lag >= 1000000 on group minikube.default.london-keyword-count and topic reddit_posts",
    "bdb01792-b79f-44d0-a39e-7f3e0906a89b": "lag >= 100000 on group connect-nullsink and topic topicA",
    "f4dd550f-bcb6-4e02-b28b-18bbd426aaf2": "lag >= 100000 on group groupA and topic topicA"
}

Create or update an existing condition for a specific Alert Setting

$ lenses-cli alert setting condition set \
--alert=2000 \
--condition="lag >= 200000 on group groupA and topic topicA"

Update condition from a file:

$ lenses-cli alert setting condition set ./alert_cond.yaml

alert_cond.yaml:

Alert: 2000
Condition: >
lag >= 200000 on
group groupA and topic topicA

Note

Command line flags are ignored when loading from files

Delete an existing condition for a specific Alert Setting

Note

The --alert flag is the Alert ID that the alert setting belongs to, the --condition flag is the UUID of the condition to delete

TroubleShooting¶

For troubleshooting or additional information and join our slack channel