Bring your own Application

Introduction 

You can link any custom microservice or other streaming application running and producing, consuming or processing data in Kafka by self-registering the application. This will make it appear in the apps catalog, monitor and visualize it in your topology. There are two ways you can do that:

• By using a REST endpoint to register some metadata, runners, lineage information etc.
• or if you have a JVM based application, there is a JVM client you can use to instrument the app

Register via REST endpoint 

Here is a video overview on how it works:

Required permission 

Access Management & permissions

App state and Runners 

Each app can be registered with a set of endpoints that return the status of each app’s running instances. These endpoints are called runners and allow Lenses to get the running instances’ status by pinging these endpoints. The app state is being consolidated based on the individual runner’s state, which are periodically checked.

Each runner can be only:

• RUNNING - When the health-check endpoints return 200 HTTP Status Code
• UNKNOWN - When the health-check endpoints return anything other than 200 HTTP Status Code

Lenses grabs and consolidates the individual runners’ statuses to a single app state as it follows:

• RUNNING - All Runners are RUNNING
• WARNING - At least 1 Runner is UNKWOWN
• UNKNOWN - No information about Runners, either they are unreachable

Register new Application 

To add your external apps to Lenses, you can currently use the API by doing a POST to api/v1/apps/external endpoint.

curl -X POST \
  <LENSES_HOST_URL>/api/v1/apps/external \
  -H 'Content-Type: application/json' \
  -H 'X-Kafka-Lenses-Token: <LENSES_AUTH_TOKEN>' \
  -d '{
    "name": "Example_Producer_App",
    "metadata": {
        "version": "1.0.0",
        "owner": "Lenses",
        "deployment": "K8s",
        "tags": [
            "fraud",
            "detection",
            "app"
        ]
    },
    "input": [{"name": "fraud_input_topic"}],
    "output": [{"name": "fraud_output_topic"}],
    "runners": [{"url": "<YOUR_HEALTH_CHECK_URL_1>", "name": "Example_Runner"}]
}'

App catalog & monitor status 

Apps are another type of built-in rules. Similar to Infrastructure ones, they can be enabled or disabled but also can be routed into any channel you configured.

In the application details page, you can see additional information about each application, and also information about each runner, along with metadata information, such as Description, and auditing information such as Created At, Created By and Modified At, Modified By.

On top of that, we are visualising your input and output data sources, that your application is using. You can see it by clicking the Data Flow tab.

Add Runners 

You can add runners with a POST request to api/v1/apps/external. The body of the new request should be the same as the first one except for the runners key. What needs to be changed is only the array of the runners key

curl -X POST \
  <LENSES_HOST_URL>/api/v1/apps/external \
  -H 'Content-Type: application/json' \
  -H 'X-Kafka-Lenses-Token: <LENSES_AUTH_TOKEN>' \
  -d '{
    "name": "Example_Producer_App",
    "metadata": {
        "version": "1.0.0",
        "owner": "Lenses",
        "deployment": "K8s",
        "tags": [
            "fraud",
            "detection",
            "app"
        ]
    },
    "input": [{"name": "fraud_input_topic"}],
    "output": [{"name": "fraud_output_topic"}],
    "runners": [{"url": "<YOUR_HEALTH_CHECK_URL_2>", "name": "Example_Runner"}]
}'

Remove Runners 

You can also reduce the number of runners with a DELETE request to /api/v1/apps/external/{name}/runners.

curl -X DELETE <LENSES_HOST_URL>/api/v1/apps/external/Example_Producer_App/runners \
-H 'Content-Type: application/json' \
-H 'X-Kafka-Lenses-Token: <LENSES_AUTH_TOKEN>' \
-d '{ "runners":["<YOUR_HEALTH_CHECK_URL_1>", "<YOUR_HEALTH_CHECK_URL_2>"]}'

Remove your Application 

You can remove the app by making a DELETE request to /api/v1/apps/external/{name}. By Removing an App from Lenses, you just drop Lenses’ visibility for that specific App.

curl -X DELETE <LENSES_HOST_URL>/api/v1/apps/external/Example_Producer_App \
-H 'Content-Type: application/json' \
-H 'X-Kafka-Lenses-Token: <LENSES_AUTH_TOKEN>'

Alternatively, you can use the Lenses UI to remove the App:

Register via JVM client 

Lenses also supports metrics for Kafka Streams, Akka Streams, and Spark Structured Streaming. You can find full working examples on GitHub . To add your external apps to Lenses Topology through the topology client, instrument your code using the Lenses client library.

The open-source topology-client provides a simple builder for topologies, which is then pushed to a Kafka topic to be picked up by Lenses at runtime.

JVM Required permission 

Access Management & permissions

View in Topology 

Any registered application can be viewed as part of the overall topology and see where it fits in the streaming pipeline. For applications registered via the JVM we can also view the collected metrics:

API documentation 

API