Configuring the Apache Kafka Monitoring Gateway
The tibkmongateway is configured using a JSON file which is specified as part of the command line parameters to the monitor-start.py script.
This file contains all of the information needed to access Apache Kafka statistics including:
- Apache Kafka Broker and Apache ZooKeeper addresses and ports
- Apache Kafka Broker user names and passwords for access
- Specific Apache Kafka and Apache ZooKeeper statistics to collect
- InfluxDB database access statistic storage information
The JSON file itself is quite complex and repetitive so to simplify the configuration of tibkmongateway, a .csv (comma separated value) spreadsheet template is provided which can be used to programmatically generate the corresponding JSON configuration file.
As an administrator you edit this file directly (using any available spreadsheet applications or directly using a text editor if you are familiar with the CSV format), set key configuration parameters, save the file (as a .csv), and specify this file on the command line of monitor-start.py. This will generate the complete JSON file using the information provided in the .csv.
The sample .csv file is named tibkmon_config.csv and is located in the monitoring/tibkmon_JSON_Generator directory.
The .csv file has several columns each representing a specific configuration setting which will be set in the JSON file as needed in every location it is required:
| Configuration Setting | Description |
|---|---|
| Kafka Broker JMX Host & Port | The network address and host from where a specific broker's statistics can be accessed via JMX. This access must also be enabled using the Kafka Broker's configuration. |
| Kafka Broker JMX Username & Password | User authentication information for connecting to the Kafka Brokers. |
| Kafka Cluster ID | A text string which will be added to all statistics collected from this specific Kafka Broker to uniquely identify the Kafka broker cluster that this Kafka Broker is part of in the InfluxDB database.
It is recommended that this ID be unique across all your Kafka clusters. |
| Kafka Broker ID | A text string which will be added to all statistics collected from this specific Kafka Broker to uniquely identify information from it in the InfluxDB database.
It is recommended that this ID be unique across all your Kafka Brokers. |
| Enable SSL Flag | Use SSL to secure the communications between the Kafka Broker and
tibkmongateway.
Valid values are TRUE or FALSE. |
| Monitoring Gateway Number of Thread | The number of threads that
tibkmongateway should use when accessing a specific broker.
The default value is 2. Increasing this value will reduce the amount of time needed to retrieve statistics from a specific Kafka Broker and make those statistics more closely aligned in terms of access time. Increasing this number places a higher load on the Kafka Broker while decreasing it forces statistics to be gathered serially rather than concurrently. |
| InfluxDB Access URL, Database Name, User Name, & Password | The configuration settings for accessing the appropriate Influx DB instance and database to store the statistics collected from the Kafka Brokers. |
| Server Type | The value should be
Broker if this line is for an Apache Kafka Broker or
ZooKeeper if this line represents an Apache ZooKeeper Node.
If Broker is specified, the ZooKeeper HTTP Port is not required to be populated. If ZooKeeper is specified, Kafka Cluster ID and Kafka Broker ID are not required to be populated but the Enable SSL Flag field must be set to FALSE (ZooKeeper does not support SSL connections at this time). |
| ZooKeeper HTTP Port | Port to access Apache ZooKeeper statistics. |
| Alias |
A text string which will be added to all statistics collected from this specific entity to uniquely identify information from it in the InfluxDB database. It is recommended that this ID be unique across all your Kafka Brokers and ZooKeeper nodes. |