Lenses Setup Wizard improves users first touch experience through a guided installation wizard. The wizard helps you create the required connection to Kafka and optionaly configure JMX.
Enter you broker bootstrap address, security protocol and mechansims and test your connection. If successful, enter your license and Lenses will boot. You can login with admin/admin.
admin/admin
Kafka Connection contains the configuration of Kafka cluster used by Lenses. All usages of Kafka across whole Lenses application are backed by Kafka Connection.
Lenses supports all security protocols of Kafka. When it comes to security mechanisms for SASL authentication the most common are well supported.
At startup if no Kafka connection is found the setup wizard is launched. You can either configure the Kafka connection, including JMX from there or use the provision CLI command.
To edit a connection, go to:
Admin / Connections
select the Kafka connection and in actions select edit.:
Like other core services, Kafka is managed via Connections. See more about managing connections here.
Multiple connections allowed: NO
Deletion allowed: NO
When managing connections other than with GUI, such information are relevant for Kafka connection:
Template name: Kafka
Constant connection name: kafka
The easiest way to manage Connections is via Lenses GUI under their respective pages, however it is also possible to do it directly via API, Helm or Lenses CLI. In such case, some connection type-specific values have to be used. Here are few examples of such configuration in YAML format.
The only required fields are:
kafkaBootstrapServers
protocol
In following example JMX metrics for Kafka Brokers are configured too, assuming that all brokers expose their JMX metrics using the same port (9581), without SSL and authentication. Other options of configuring metrics can be found in Services Metrics.
kafka: tags: ["optional-tag"] templateName: Kafka configurationObject: kafkaBootstrapServers: - PLAINTEXT://your.kafka.broker.0:9092 - PLAINTEXT://your.kafka.broker.1:9092 protocol: PLAINTEXT # all metrics properties are optional metricsPort: 9581 metricsType: JMX metricsSsl: false
A truststore (with password) might need to be set explicitly if the global truststore of Lenses does not include the Certificate Authority (CA) of the brokers.
If TLS is used for authentication to the brokers in addition to encryption-in-transit, a keystore (with passwords) is required.
kafka: tags: ["optional-tag"] templateName: Kafka configurationObject: kafkaBootstrapServers: - SSL://your.kafka.broker.0:9093 - SSL://your.kafka.broker.1:9093 protocol: SSL sslTruststore: fileRef: filePath: /path/to/truststore.jks sslTruststorePassword: truststorePassword sslKeystore: fileRef: filePath: /path/to/keystore.jks sslKeyPassword: keyPassword sslKeystorePassword: keystorePassword
There are 2 SASL-based protocols to access Kafka Brokers: SASL_SSL and SASL_PLAINTEXT. They both require SASL mechanism and Jaas Configuration values.
SASL_SSL
SASL_PLAINTEXT
In addition to this, there might be a keytab file required, depending on the SASL mechanism (for example when using GSSAPI mechanism, most often used for Kerberos).
In order to use Kerberos authentication, a Kerberos Connection should be created beforehand. See more
Apart from that, when encryption-in-transit is used (with SASL_SSL), a truststore might need to be set explicitly if the global truststore of Lenses does not include the CA of the brokers.
Following are few examples of SASL_PLAINTEXT and SASL_SSL with misc SASL mechanisms being used.
kafka: tags: ["optional-tag"] templateName: Kafka configurationObject: kafkaBootstrapServers: - SASL_SSL://your.kafka.broker.0:9093 - SASL_SSL://your.kafka.broker.1:9093 protocol: SASL_SSL sslTruststore: fileRef: filePath: /path/to/truststore.jks sslTruststorePassword: truststorePassword saslMechanism: PLAIN saslJaasConfig: | org.apache.kafka.common.security.plain.PlainLoginModule required username="your-username" password="your-password";
kafka: tags: ["optional-tag"] templateName: Kafka configurationObject: kafkaBootstrapServers: - SASL_SSL://your.kafka.broker.0:9093 - SASL_SSL://your.kafka.broker.1:9093 protocol: SASL_SSL saslMechanism: GSSAPI sslTruststore: fileRef: filePath: /path/to/truststore.jks sslTruststorePassword: truststorePassword saslJaasConfig: | com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true useTicketCache=false serviceName=kafka principal="my-principal@DOMAIN.COM"; keytab: fileRef: filePath: /path/to/keytab.jks
The set of properties required for SCRAM-SHA-512 mechanism is the same as for SCRAM-SHA-256 mechanism.
SCRAM-SHA-512
SCRAM-SHA-256
kafka: tags: ["optional-tag"] templateName: Kafka configurationObject: kafkaBootstrapServers: - SASL_PLAINTEXT://your.kafka.broker.0:9093 - SASL_PLAINTEXT://your.kafka.broker.1:9093 protocol: SASL_PLAINTEXT saslMechanism: SCRAM-SHA-256 saslJaasConfig: | org.apache.kafka.common.security.scram.ScramLoginModule required username="your-username" password="your-password";
When Lenses is running inside AWS and is connecting to an Amazon’s Managed Kafka (MSK) instance, IAM can be used for authentication.
kafka: tags: ["optional-tag"] templateName: Kafka configurationObject: kafkaBootstrapServers: - SASL_SSL://your.kafka.broker.0:9098 - SASL_SSL://your.kafka.broker.1:9098 protocol: SASL_SSL saslMechanism: AWS_MSK_IAM saslJaasConfig: | software.amazon.msk.auth.iam.IAMLoginModule required; additionalProperties: sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler" metricsType: AWS # Settings below are optional metricsUsername: "your-username" metricsPassword: "your-password" metricsSsl: true metricsHttpTimeout: 30000 metricsPort: 11001
Lenses interacts with your Kafka Cluster via Kafka Client API. To override the default behavior use additionalProperties.
additionalProperties
By default there shouldn’t be a need to use additional properties, use it only if really necessary, as a wrong usage might brake the communication with Kafka.
Lenses SQL processors uses the same Kafka connection information provided to Lenses.
kafka: tags: ["optional-tag"] templateName: Kafka configurationObject: kafkaBootstrapServers: - PLAINTEXT://your.kafka.broker.0:9092 protocol: PLAINTEXT additionalProperties: isolation.level: "read_committed" acks: "all" ssl.endpoint.identification.algorithm: "https"
Find more about multiple options of configuring services’ metrics (like secured JMX, Jolokia, etc) under Services Metrics
Lenses Wizard appears when we have a running Lenses instance with no configured Kafka Brokers.
There are 3 mandatory steps:
In the Last step we validate the configuration details and if no issues found full lenses experience is available.
The user inputted information remains to the wizard as long as the browser tab remains open. Lenses does not store any information till the wizard is finished.
When wizard is finished a user can visit Admin -> Connections -> Kafka page and amend the Broker Metrics information. The changes apply on save and no reboot of Lenses is needed
On this page
