Creating a Prometheus Metric Job

A metric job is used to configure a data source for Prometheus metrics exporter to scrape metrics.

Note: Ensure that you have specified the Domain and Agent parameters correctly in the New Metric window while configuring a data source.

Creating a Prometheus Metric Job for Kubernetes Service Discovery

If you want the targets to be automatically discovered in the Kubernetes cluster for the Prometheus exporter to scrape the metrics, then perform the following steps:

Note: You can choose this option only if the Hawk agent is deployed in a Kubernetes cluster.
1. Navigate to Exporters > Prometheus Metrics page.
2. In the Prometheus Metrics page, the jobs for the Prometheus Metrics are displayed.
3. Click the Add Prometheus Metric icon () to open the New Metric window.
4. Specify the following parameters in the left panel of the New Metric window:
— Domain - Name of the domain. Select the name of the domain from the drop down menu.
— Agent - Name of the agent. Only the agents belonging to the specified Domain are available to be selected from the drop down menu.
— Job Name - Name of the job. A microagent is created based on the job name. A job name should be intuitive and ideally hint at the nature of the job. The job name must not contain any whitespaces.
— Metrics Path - The HTTP API path from which the Prometheus metrics are scraped.
— Scheme - Protocol to be used for communication.
— Honor Labels - Specifies whether labels like job and instance should be kept from scraped data.
— Authorization Type - Select the type of authorization required. This will specify the authorization header that is to be used. For No Auth, you must provide any information. For Basic Auth, you need to provide a user name and a password. For Bearer Token, you need to provide a Bearer Token. If you toggle Use a Token File for Bearer Token, you need to provide a path pointing to the Bearer Token file.
Note: The bearer token file and the Hawk Agent must be in the same container/machine/VM.
— TLS - If TLS is used, you need to specify the path of the Server CA Certificate, client Certificate, client Private Key files and Server name.
Note: The files must be present in the Hawk Agent container/machine/VM and the path must be provided with respect to this container.
— Insecure Skip Verify - Disables validation of the server certificate.
5. Select Kubernetes Service Discovery in the right panel of New Metric window and then specify the following parameters:
— Role - Select one of the following roles to discover targets. Each role has a meta label associated with it. The available roles are Service, Pod, Node, and Endpoints.
Service - The targets are discovered from each service port of Kubernetes service. The default address is Kubernetes DNS name of the service and the service port.
Pod - The targets are discovered from pod containers. A target is discovered for each declared port of a container.
Node - The targets are created for each node present in the Kubernetes cluster. The target address defaults to the first existing address of the Kubernetes node object.
Endpoints - The targets are discovered from endpoints of Kubernetes service. For each port in an endpoint address, a target is discovered.
6. (Optional) You can add relabel configuration. Relabel configuration enables you to add meta labels into final metrics and also helps in deciding whether to keep or drop a particular Kubernetes resource from scraping. Add a relabel configuration by clicking the Add Relabel Configuration icon (), then click on the expand icon () beside the configuration to expand the configuration pane and then specify the following parameters:
Note: You can add more than one relabel configuration.
— Add Source Label - Click the Add Source Label button to add a source label. The source labels select values from existing meta labels. Their content is concatenated using the configured separator and matched against the configured regular expression for the replace, keep, and drop actions.
— Separator - A separator is placed between concatenated source label values. The default value is a semicolon (;).
— Replacement - It is a value with which a regex replace is performed after the regular expression matches with a label. The default value is $1.
— Regex - It is a regular expression with which the extracted value is matched. The default value is (.*).
— Target Label - It is a label to which the replace value is written. It is mandatory to define this value for replace action.
— Select one of the Actions depending on which an operation is performed if regular expression is matched:
Replace - Matches regex with the concatenated source_labels and then sets target_label for replacement with match group references substituted by their value. No replacement operation is performed if regular expression does not match.
Keep - Drop the targets for which regex does not match with the concatenated source_labels.
Drop - Drops the targets for which regex matches with the concatenated source_labels.
Labelmap - Matches regex with all the label names then copies the values of the matching labels to label names given by replacement with match group references (${1}, ${2}, ...) in replacement substituted by their value.
Labelkeep - Matches regex with all label names. Any label that does not match with the regex will be removed from the set of labels.
Labeldrop - Matches regex with all label names. Any label that matches with the regex will be removed from the set of labels.
7. Click Validate to validate whether the specified parameters produce an output.
8. Click Create Metric.

Creating a Prometheus Metric Job for Static Configuration

If you want to manually specify the target for the Prometheus exporter to scrape the metrics, then perform the following steps:

Procedure 

1. Navigate to Exporters > Prometheus Metrics page.
2. In the Prometheus Metrics page, the jobs for the Prometheus Metrics are displayed.
3. Click the New Metric icon () to open the New Metric window.
4. Specify the following parameters in the left panel of the New Metric window:
— Domain - Name of the domain. Select the name of the domain from the drop down menu.
— Agent - Name of the agent. Only the agents belonging to the specified Domain are available to be selected from the drop down menu.
— Job Name - Name of the job. A microagent will be created based on the job name. A job name should be intuitive and ideally hint at the nature of the job. The job name must not contain any whitespaces.
— Metrics Path - The HTTP API path from which the Prometheus metrics will be scraped.
— Scheme - Protocol to be used for communication.
— Honor Labels - Specifies whether labels like job and instance should be kept from scraped data.
— Authorization Type - Select the type of authorization required. This will specify the Authorization header that is to be used. For No Auth, you do not need to provide any information. For Basic Auth, you need to provide a user name and a password. For Bearer Token, you need to provide a Bearer Token. If you enable Use a Token File for Bearer Token, you need to provide a path pointing to the Bearer Token file.
Note: The bearer token file and the Hawk Agent must be in the same container/machine/VM.
— TLS - If TLS is used, you need to specify the path of the Server CA Certificate, client Certificate, client Private Key files and Server name.
Note: The files must be present in the Hawk Agent container/machine/VM and the path must be provided with respect to this container.
— Insecure Skip Verify - Disables validation of the server certificate.
5. Select Static Configuration in the right panel of New Metric window and then specify the following parameters:
— Targets - Enter the target in the following format: <Target_name>:Port. You can add more than one target for scraping data.
— Labels - This is optional. The user can add labels for a target by clicking the add icon () and then specifying the name and value of the label. These labels will be added in the metrics scraped from the specified targets.
6. Click Validate to validate whether the specified parameters produce an output.
7. Click Create Metric.