Configuration and Script Examples

To verify that your TIBCO Messaging - Bridge for Apache KafkaMonitoring installation is ready for configuration and deployment, perform the following general tasks and confirmations. This topic contains simple default configurations and usage examples; your requirements may vary. Before you start Apache Kafka monitoring, you can convert your older .csv file, or you can make changes directly to the sample .yaml file.

Prerequisites

  • Ensure that you have Python 3 installed before you begin.
  • Run the following script to install any Python dependencies required by Apache Kafka Monitoring: 
    pip3 install -r /opt/tibco/akd/bridge/2.3/monitoring/requirements.txt

Edit the config/kafka_config.yaml file

Begin by editing the kafka_config.yaml file, which is located in /opt/tibco/akd/bridge/2.3/monitoring/config

This file contains configuration information pertaining to the Apache Kafka cluster to be monitored. Edit this file to enter specifics about your Apache Kafka Cluster and Apache ZooKeeper nodes. You may need to modify the following fields: 

settings:
    grafana:
        url, username, password
        datasource:
            type, access, basicAuth, isDefault

    influxDB:
        url, username, password

The default configuration in infrastructure_config.yaml should be sufficient to start the InfluxDB and Grafana services.

Note: The configured maximum number of file handlers must be high enough for InfluxDB to monitor multiple Apache Kafka services in parallel. You can see the file handlers limit by executing "ulimit -n". You can set the file handlers limit by executing "ulimit -n <int>".

Start monitoring using the following procedure:

  1. Change to the monitoring directory: 
    cd /opt/tibco/akd/bridge/2.3/monitoring
  2. Run the start script. 
    bin/tibkmon-start.py -kc config/kafka_config.yaml -ic config/infrastructure_config.yaml

    You see an output similar to the following: 

    TIBCO Apache Kafka Monitoring: 2.3.0 V8

*************************************
 INFO    [argument_parser.py:160] TIBKMON logging level set to INFO
 INFO    [measurements.py:67] Successfully loaded all templates and drivers.
 INFO    [validate.py:36] Validation successful. Infrastructure-config @/opt/tibco/akd/bridge/2.3/monitoring/config/infrastructure_config.yaml
 INFO    [validate.py:322] Validation successful. Kafka-config @/opt/tibco/akd/bridge/2.3/monitoring/config/kafka_config.yaml
 INFO    [generator.py:321] zookeeper_one..zookeepers..mon-remote-qa added to list of jmxTrans_Servers
 INFO    [generator.py:321] zookeeper_two..zookeepers..mon-remote-qa added to list of jmxTrans_Servers

*
*
*
 INFO    [loader.py:88] Successfully posted grafana-dashboard @ /opt/tibco/akd/bridge/2.3/monitoring/tibkmon/lib/../../tibkmon/grafana-dashboards/InfluxDB-kafka-producers.json
 INFO    [loader.py:89] Grafana Dashboard HTTP-Post Response: {"id":10,"slug":"kafka-producers","status":"success","uid":"8Qub_QhMk","url":"/d/8Qub_QhMk/kafka-producers","version":1}
------------------------------------------------------------
Note: To see all available options, use the -h option.

To check that the monitoring for all of the cluster elements has started, enter: 

bin/tibkmon-status.py

You see an output similar to the following: 

TIBCO Apache Kafka Monitoring: 2.3.0 V8

*************************************

 INFO    [tibkmon-status.py:157] TIBKMON logging level set to INFO
 INFO    [tibkmon-status.py:94] Status File loaded from: /var/tmp/<user>/tibco-tibkmon/status/tibkmon.yaml
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Grafana Status : Running

  jmxTrans Status : Running

  InfluxDB Status : Running

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To see detailed information about the status of InfluxDB, Grafana, jmxTrans and Apache Kafka clusters being monitored, use the command using the --detailed option as follows: 
bin/tibkmon-status.py --detailed.
To stop monitoring of your Apache Kafka cluster, run the stop script: 
bin/tibkmon-stop.py

You see an output similar to the following: 

TIBCO Apache Kafka Monitoring: 2.3.0 V8

*************************************
 INFO    [tibkmon-stop.py:213] TIBKMON logging level set to INFO
 INFO    [tibkmon-stop.py:117] Status File loaded from: /var/tmp/<user>/tibco-tibkmon/status/tibkmon.yaml
 INFO    [tibkmon-stop.py:125] Stopped process jmxTrans running with pid: 61058
 INFO    [tibkmon-stop.py:146] Stopped process Grafana running with pid: 61047
 INFO    [tibkmon-stop.py:163] Stopped process InfluxDB running with pid: 61046
 INFO    [tibkmon-stop.py:181] Status File written to : /var/tmp/<user>/tibco-tibkmon/status/tibkmon.yam

To convert your existing .csv-based configuration information to the new .yaml format:

  1. Change to the monitoring directory: 
    cd /opt/tibco/akd/bridge/2.3/monitoring
  2. Run the following command: 
    bin/tibkmon-converter.py -f tibkmon.csv -o .
Example output: 
*************************************

TIBCO Apache Kafka Monitoring: TIBCO Apache Kafka Monitoring: 2.3.0 V8

*************************************
 INFO    [tibkmon-converter.py:369] TIBKMON logging level set to INFO
 INFO    [tibkmon-converter.py:329] YAML file is saved at:/Users/xyz/tmp/tibkmon-converter.yaml

The monitoring configuration information in the .csv file is now in the new .yaml file. The file has sections for each Apache Kafka broker and Apache ZooKeeper node. You can use this file with tibkmon-start.py using the -kc option.