Apache Kafka Monitoring

The tibkmongateway process polls a configured set of Apache Kafka-related services like Brokers, ZooKeepers, Connectors, Streams, Producers & Consumers. It saves the polled information to an InfluxDB database for retrieval, display, and analysis using customizable Grafana-based dashboards.

For ease of configuration, a .yaml file is used to configure the monitoring behavior, allowing easy configuration for each Apache Kafka component, including jmx-host and jmx-port settings, security settings, and other component-specific settings.

Note: Earlier releases used a .csv file (comma-separated value) to store monitoring-related configuration information. When upgrading to 2.3 you can migrate your existing .csv-based configuration information into the new .yaml format. For information on converting .csv files to .yaml files using the tibkmon-converter.py script, see the information in Configuration Example. If you choose to migrate your existing information, you should perform the migration before you begin monitoring.

Scope

The TIBCO Apache Kafka Monitoring library gathers monitoring data from the following Apache Kafka components (the default sampling rate is every 60 seconds, but this value can be modified):
  • Apache Kafka Brokers
  • Apache Kafka-related Apache ZooKeeper information
  • Apache Kafka clients
  • Apache Kafka stream modules
  • Apache Kafka Connectors