Changing Output Destinations for Different Types of Logs using CLI

This topic contains information about how to configure output channels to send logs, access logs and metrics to Database, Forward, Elasticsearch, HTTP, Kafka, Syslog, generic TCP end points using command-line interface.

How to Use This Feature

This feature allows you to change the output destination for different types of logs in one invocation.

Caution: Using the CLI overwrites the existing configuration. Therefore, each time it is used, the system expects to receive a full configuration. Incomplete configurations are ignored when generating the log service's configurations.

The command accepts a JSON that contains the different configuration properties. The data in the file should adhere to JSON rules. All properties are top-level attributes. See the examples below.

Command Line

The generic form of the command is:

clustermanager import config --componentType <component type> [-- componentId <componentId>] --file <full path to json>

For example: 

cm 
import
 config [--componentId 3a076250-955b-49ef-ab06-ea19df10a665 --componentType logservice --file props.json

Types of Logs

You can export the following types of logs to the chosen destination:
  1. Access logs
  2. Metrics
  3. Container logs
  4. Verbose logs

Each type of log has a different set of property names. This way you can send different types of logs to different destinations.

The logs are written to the disk (DEFAULT) on the log service container irrespective of the destination you choose.

All values and property names are in character/string data type. Depending on chosen output channel type, the remaining configuration will change.
Note: To configure log service to communicate with Elasticsearch via HTTPS, copy the CA certificate (in PEM form) of the elastic search host to /docker-deploy/resources. The files are in the log container.
Access Logs
Destination Type Name in JSON Description Notes Example
td_agent_output_channelType Comma separated string of types of channels Applicable values are DEFAULT, DATABASE, FORWARD, ELASTICSEARCH, HTTP, KAFKA, SYSLOG, TCP. "td_agent_output_channelType": "<Destination Type>"
DATABASE td_agent_out_dbType Type of database to be connected. Applicable values are MYSQL, POSTGRESQL "td_agent_out_dbType":"MYSQL"

OR

"td_agent_out_dbType":"POSTGRESQL"

td_agent_out_dbHost DBHost where Database instance is running. The hostname or IP should be reachable from the log service container "td_agent_out_dbHost":"MYDBHOST"

OR

"td_agent_out_dbHost":"192.168.1.11"

td_agent_out_dbPort A port on which the database is listening. "td_agent_out_dbPort":"3306"

OR

"td_agent_out_dbPort":"5432"

td_agent_out_dbSchema Database schema under which access_log table is created or existed. "td_agent_out_dbSchema":"masherylocallogs"
td_agent_out_dbUser The database user with permission to access. "td_agent_out_dbUser":"DBUSER"
td_agent_out_dbPassword Password for given user. "td_agent_out_dbPassword":"PASSWORD"
FORWARD td_agent_out_forward_servers Hostname or IP address and port for available FluentD servers.

Expected value is [{HOST}:{PORT},{HOST1}:{PORT}:{HOST2}:{PORT}]

The hostname should be reachable from the log service container.

LogService service will push data to any of the available servers.

  • With port:
    • "td_agent_out_forward_servers": 192.168.1.11:24221,192.168.2.11:24225"
    • "td_agent_out_forward_servers" : fluentHost1:24221,fluentHost2:24225"
  • Without port: "td_agent_out_forward_servers" : fluentHost1,fluentHost2".

    Default port value: 24224

ELASTICSEARCH td_agent_out_elasticsearch_host Host name or IP address of Elastic search server or ELB. The hostname should be reachable from within the log service container. "td_agent_out_elasticsearch_host" : "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_elasticsearch_port "td_agent_out_elasticsearch_port" : "9200"
td_agent_out_elasticsearch_index "td_agent_out_elasticsearch_index" : "ml5_access_logs"
td_agent_out_elasticsearch_protocol The protocols supported are HTTP and HTTPS. This is optional. The default protocol is HTTP. "td_agent_out_elasticsearch_protocol":"https"
td_agent_out_elasticsearch_ssl_version SSL version configured on the destination elastics search instance. Use this if td_agent_out_elasticsearch_protocol is HTTPS. Supported versions are TLSv1_2, TLSv1, TLSv1_1,SSLv23. Default version is TLSv1_2. "td_agent_out_elasticsearch_ssl_version":"TLSv1_2"
td_agent_out_elasticsearch_cafile The Name of CA file to be used. Use this if td_agent_out_elasticsearch_protocol is HTTPS. The value should be the name of the file. "td_agent_out_elasticsearch_cafile":"cafile-a06ebe8a15a3011e9bf380608b2-531419666.us-east-1.elb.amazonaws.com.crt"
td_agent_out_elasticsearch_user User name to authenticate. This is optional depending on the elastic search security settings. " td_agent_out_elasticsearch_user":"elasticuser"
td_agent_out_elasticsearch_password Credential "td_agent_out_elasticsearch_password":"password123"
HTTP td_agent_out_http_URI HTTP URI to which logs are to be sent. The HTTP URI should be accessible from the log container.
KAFKA td_agent_out_kafka_brokers Comma delimited list of kafkabroker_host:port. The Kafka brokers host should be reachable from the log service and broker should be running. "td_agent_out_kafka_brokers":"broker1:9092,broker2:9092"
td_agent_out_kafka_topic "td_agent_out_kafka_topic" : "ml5_access_logs"
SYSLOG td_agent_out_syslog_host Host name of IP address where syslog is running. Host should be reachable from log service. "td_agent_out_syslog_host": "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_syslog_port Port number of syslog program. "td_agent_out_syslog_port":"5142"
td_agent_out_syslog_tag Tag with which the log is to be uniquely identifed. Defaults to incoming log event tag. "td_agent_out_syslog_tag":"ml5_access_logs"
td_agent_out_syslog_protocol Protocol on which syslog is listening for incoming requests. Defaults to tcp. "td_agent_out_syslog_protocol":"tcp"
TCP td_agent_out_tcp_host Host name where a TCP socket is listening. Host should be reachable from log service "td_agent_out_tcp_host": "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_tcp_port Port on which the TCP socket is listening. "td_agent_out_tcp_port":"6000"
Metrics Log
Destination Type Name in JSON Description Notes Example
td_agent_metric_output_channelType Comma separated string of types of channels Applicable values are FORWARD, ELASTICSEARCH, HTTP, KAFKA, SYSLOG,TCP "td_agent_metric_output_channelType": "<Destination Type>"
FORWARD td_agent_out_metric_forward_servers Hostname or IP address and port for available FluentD servers

Expected value is [{HOST}:{PORT},{HOST1}:{PORT}:{HOST2}:{PORT}]

The hostname should be reachable from the log service container.

LogService service will push data to any of the available servers.

  • With port:
    • "td_agent_out_metric_forward_servers": 192.168.1.11:24221,192.168.2.11:24225"
    • "td_agent_out_metric_forward_servers" : fluentHost1:24221,fluentHost2:24225"
  • Without port: "td_agent_out_metric_forward_servers" : fluentHost1,fluentHost2".

    Default port value: 24224

ELASTICSEARCH td_agent_out_metric_elasticsearch_host Host name or IP address of Elastic search server or ELB The hostname should be reachable from within the log service container "td_agent_out_metric_elasticsearch_host" : "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_metric_elasticsearch_port "td_agent_out_metric_elasticsearch_port" : "9200"
td_agent_out_metric_elasticsearch_index Defaults to incoming log event tag. "td_agent_out_metric_elasticsearch_index" : "ml5_metric_logs"
td_agent_out_metric_elasticsearch_protocol The protocols supported are HTTP and HTTPS. This is optional. The default protocol is HTTP. "td_agent_out_metric_elasticsearch_protocol":"https"
td_agent_out_metric_elasticsearch_ssl_version SSL version configured on the destination elastics search instance. Use this if td_agent_out_elasticsearch_protocol is HTTPS. Supported versions are TLSv1_2, TLSv1, TLSv1_1,SSLv23. Default version is TLSv1_2. "td_agent_out_metric_elasticsearch_ssl_version":"TLSv1_2"
td_agent_out_metric_elasticsearch_cafile The Name of CA file to be used. Use this if td_agent_out_elasticsearch_protocol is HTTPS. The value should be the name of the file. "td_agent_out_metric_elasticsearch_cafile":"cafile-a06ebe8a15a3011e9bf380608b2-531419666.us-east-1.elb.amazonaws.com.crt"
td_agent_out_metric_elasticsearch_user User name to authenticate. This is optional depending on the elastic search security settings. " td_agent_out_metric_elasticsearch_user":"elasticuser"
td_agent_out_metric_elasticsearch_password Credential " td_agent_out_metric_elasticsearch_password":"password123"
HTTP td_agent_out_metric_http_URI HTTP URI to which logs are to be sent. The HTTP URI should be accessible from the log container "td_agent_out_metric_http_URI": "http://a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com /ml5/metriclogs"
KAFKA td_agent_out_metric_kafka_brokers Comma separated list of kafkabroker_host:port. The Kafka brokers host should be reachable from the log service and broker should be running. "td_agent_out_metric_kafka_brokers":"broker1:9092,broker2:9092"
td_agent_out_metric_kafka_topic "td_agent_out_metric_kafka_topic" : "ml5_metric_logs"
SYSLOG td_agent_out_metric_syslog_host Host name of IP address where syslog is running. Host should be reachable from log service. "td_agent_out_metric_syslog_host": "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_metric_syslog_port Port number of syslog program. "td_agent_out_metric_syslog_port":"5142"
td_agent_out_metric_syslog_tag Tag with which the log is to be uniquely identifed. Defaults to incoming log event tag. "td_agent_out_metric_syslog_tag":"ml5_metric_logs"
td_agent_out_metric_syslog_protocol Protocol on which syslog is listening for incoming requests. Defaults to tcp. "td_agent_out_metric_syslog_protocol":"tcp"
TCP td_agent_out_metric_tcp_host Host name where a TCP socket is listening. Host should be reachable from log service. "td_agent_out_metric_tcp_host": "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_metric_tcp_port Port on which the TCP socket is listening. "td_agent_out_metric_tcp_port":"6000"
Container Logs
Destination Type Name in JSON Description Notes Example
td_agent_container_output_channelType Comma separated string of types of channels Applicable ones are FORWARD, ELASTICSEARCH, HTTP, KAFKA, SYSLOG,TCP "td_agent_container_output_channelType": "<Destination Type>"
FORWARD td_agent_out_container_forward_servers Hostname or IP address and port for available FluentD servers

Expected value is [{HOST}:{PORT},{HOST1}:{PORT}:{HOST2}:{PORT}]

The hostname should be reachable from the log service container.

LogService service will push data to any of the available servers.

  • With port:
    • "td_agent_out_container_forward_servers": 192.168.1.11:24221,192.168.2.11:24225"
    • "td_agent_out_container_forward_servers" : fluentHost1:24221,fluentHost2:24225"
  • Without port: "td_agent_out_container_forward_servers" : fluentHost1,fluentHost2".

    Default port value: 24224

ELASTICSEARCH td_agent_out_container_elasticsearch_host The hostname should be reachable from within the log service container "td_agent_out_container_elasticsearch_host" : "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_container_elasticsearch_port "td_agent_out_container_elasticsearch_port" : "9200"
td_agent_out_container_elasticsearch_index Defaults to incoming log event tag. "td_agent_out_container_elasticsearch_index" : "ml5_container_logs"
td_agent_out_container_elasticsearch_protocol The protocols supported are HTTP and HTTPS. This is optional. The default protocol is HTTP. "td_agent_out_container_elasticsearch_protocol":"https"
td_agent_out_container_elasticsearch_ssl_version SSL version configured on the destination elastics search instance. Use this if td_agent_out_elasticsearch_protocol is HTTPS. Supported versions are TLSv1_2, TLSv1, TLSv1_1,SSLv23. Default version is TLSv1_2. "td_agent_out_container_elasticsearch_ssl_version":"TLSv1_2"
td_agent_out_container_elasticsearch_cafile The Name of CA file to be used. Use this if td_agent_out_elasticsearch_protocol is HTTPS. The value should be the name of the file. " td_agent_out_container_elasticsearch_cafile":"cafile-a06ebe8a15a3011e9bf380608b2-531419666.us-east-1.elb.amazonaws.com.crt"
td_agent_out_container_elasticsearch_user User name to authenticate. This is optional depending on the elastic search security settings. "td_agent_out_container_elasticsearch_user":"elasticuser"
td_agent_out_container_elasticsearch_password Credential " td_agent_out_container_elasticsearch_password":"password123"
HTTP td_agent_out_container_http_URI HTTP URI to which logs are to be sent. The HTTP URI should be accessible from the log container. "td_agent_out_container_http_URI": "http:// a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com /ml5/containerlogs"
KAFKA td_agent_out_container_kafka_brokers Comma separated list of kafkabroker_host:port. The Kafka brokers host should be reachable from the log service and broker should be running. "td_agent_out_container_kafka_brokers":"broker1:9092,broker2:9092"
td_agent_out_container_kafka_topic Defaults to incoming log event tag. "td_agent_out_container_kafka_topic" : "ml5_container_logs"
SYSLOG td_agent_out_container_syslog_host Host name of IP address where syslog is running. Host should be reachable from log service. "td_agent_out_container_syslog_host": "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_container_syslog_port Port number of syslog program. "td_agent_out_container_syslog_port":"5142"
td_agent_out_container_syslog_tag Tag with which the log is to be uniquely identifed. Defaults to incoming log event tag. "td_agent_out_container_syslog_tag":"ml5_containerlogs"
td_agent_out_container_syslog_protocol Protocol on which syslog is listening for incoming requests. Defaults to TCP. "td_agent_out_container_syslog_protocol":"tcp"
TCP td_agent_out_container_tcp_host Host name where a TCP socket is listening. Host should be reachable from log service. "td_agent_out_container_tcp_host": "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_container_tcp_port Port on which the TCP socket is listening. "td_agent_out_container_tcp_port":"6000"
Verbose Logs
Destination Type Name in JSON Description Notes Example
td_agent_verbose_output_channelType Comma separated string of types of channels Applicable ones are FORWARD, ELASTICSEARCH, HTTP, KAFKA, SYSLOG,TCP "td_agent_verbose_output_channelType": "<Destination Type>"
FORWARD td_agent_out_verbose_forward_servers Hostname or IP address and port for available FluentD servers

Expected value is [{HOST}:{PORT},{HOST1}:{PORT}:{HOST2}:{PORT}]

The hostname should be reachable from the log service container.

LogService service will push data to any of the available servers.

  • With port:
    • "td_agent_out_verbose_forward_servers": 192.168.1.11:24221,192.168.2.11:24225"
    • "td_agent_out_verbose_forward_servers" : fluentHost1:24221,fluentHost2:24225"
  • Without port: "td_agent_out_verbose_forward_servers" : fluentHost1,fluentHost2".

    Default port value: 24224

ELASTICSEARCH td_agent_out_verbose_elasticsearch_host The hostname should be reachable from within the log service container "td_agent_out_verbose_elasticsearch_host" : "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_verbose_elasticsearch_port "td_agent_out_verbose_elasticsearch_port" : "9200"
td_agent_out_verbose_elasticsearch_index Defaults to incoming log event tag. "td_agent_out_verbose_elasticsearch_index" : "ml5_container_logs"
td_agent_out_verbose_elasticsearch_protocol The protocols supported are HTTP and HTTPS. This is optional. The default protocol is HTTP. "td_agent_out_verbose_elasticsearch_protocol":"https"
td_agent_out_verbose_elasticsearch_ssl_version SSL version configured on the destination elastics search instance. Use this if td_agent_out_elasticsearch_protocol is HTTPS. Supported versions are TLSv1_2, TLSv1, TLSv1_1,SSLv23. Default version is TLSv1_2. "td_agent_out_verbose_elasticsearch_ssl_version":"TLSv1_2"
td_agent_out_verbose_elasticsearch_cafile The Name of CA file to be used. Use this if td_agent_out_elasticsearch_protocol is HTTPS. The value should be the name of the file. " td_agent_out_verbose_elasticsearch_cafile":"cafile-a06ebe8a15a3011e9bf380608b2-531419666.us-east-1.elb.amazonaws.com.crt"
td_agent_out_verbose_elasticsearch_user User name to authenticate. This is optional depending on the elastic search security settings. "td_agent_out_verbose_elasticsearch_user":"elasticuser"
td_agent_out_verbose_elasticsearch_password Credential " td_agent_out_verbose_elasticsearch_password":"password123"
HTTP td_agent_out_verbose_http_URI HTTP URI to which logs are to be sent. The HTTP URI should be accessible from the log container. "td_agent_out_verbose_container_http_URI": "http:// a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com /ml5/containerlogs"
KAFKA td_agent_out_verbose_kafka_brokers Comma separated list of kafkabroker_host:port. The Kafka brokers host should be reachable from the log service and broker should be running. "td_agent_out_verbose_kafka_brokers":"broker1:9092,broker2:9092"
td_agent_out_verbose_kafka_topic Defaults to incoming log event tag. "td_agent_out_verbose_kafka_topic" : "ml5_container_logs"
SYSLOG td_agent_out_verbose_syslog_host Host name of IP address where syslog is running. Host should be reachable from log service. "td_agent_out_verbose_syslog_host": "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_verbose_syslog_port Port number of syslog program. "td_agent_out_verbose_syslog_port":"5142"
td_agent_out_verbose_syslog_tag Tag with which the log is to be uniquely identifed. Defaults to incoming log event tag. "td_agent_out_verbose_syslog_tag":"ml5_containerlogs"
td_agent_out_verbose_syslog_protocol Protocol on which syslog is listening for incoming requests. Defaults to TCP. "td_agent_out_verbose_syslog_protocol":"tcp"
TCP td_agent_out_verbose_tcp_host Host name where a TCP socket is listening. Host should be reachable from log service. "td_agent_out_verbose_tcp_host": "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_verbose_tcp_port Port on which the TCP socket is listening. "td_agent_out_verbose_tcp_port":"6000"

Importing Configurations From Cluster Manager Command Line Interface

To import configurations from cluster manager command line interface follow these steps:
  1. List components: 
    clustermanager ls components
Using cluster [tml52clusterpune1]
Using Zone [eastus]
Component ID                          Type            Name               Status   Last Heartbeat Received       Host        Service Port(s)            
------------------------------------- --------------- ----------------- -------- ----------------------------- ---------- ----------------
f41c95b9-6059-42c1-8a4f-fb2261974c14  cache           cache-set-0-0.cac... ACTIVE  Mar 12 2020 08:59:09 +0000  10.244.1.6 11212,11211,11213,11214,11215,11216
710b9df4-399a-4551-b1ed-c1fc1f8ed744  configmanager   cm-deploy-0-85fbf... ACTIVE  Mar 12 2020 08:59:25 +0000  10.244.2.6 8080                      
c1057ef0-ab4e-459f-8413-0e5036c65b7c  logservice      log-set-0-0.log-s... ACTIVE  Mar 12 2020 08:59:30 +0000  10.244.2.7 24224                     
d516fe9e-3ac3-4a09-a9b0-1dceba36a23d  nosql           cass-set-0-2.cass... ACTIVE  Mar 12 2020 08:59:09 +0000  10.244.1.5 9042                      
29812c89-2e87-4dc0-a981-890cd1a72933  nosql           cass-set-0-0.cass... ACTIVE  Mar 12 2020 08:59:09 +0000  10.244.2.5 9042                      
99950720-6ded-46be-a87f-dc82be47f265  nosql           cass-set-0-1.cass... ACTIVE  Mar 12 2020 08:59:09 +0000  10.244.0.6 9042                      
8a5a78f0-f032-4046-ba59-28a75004c9ce  sql             mysql-set-0-0.mys... ACTIVE  Mar 12 2020 08:59:34 +0000  10.244.2.8 3306                      
111628dc-7893-45c8-8add-8a7944a2ff8c  trafficmanager  tm-deploy-0-5f787... ACTIVE  Mar 12 2020 08:59:28 +0000  10.244.0.7 8080
  2. Get the component ID of logservice from the output.
  3. Check the existing configuration. 
    clustermanager ls config --componentId c1057ef0-ab4e-459f-8413-0e5036c65b7c --componentType logservice
clusterId [ad41097e-35bc-4fc4-bb22-084dc25fdbf2] and zoneId [9046178a-00f9-4d59-a92e-59824142df56]
No settings found for the given componentId [c1057ef0-ab4e-459f-8413-0e5036c65b7c] of type [logservice]
  4. Create a JSON file for input (for example,/home/builder/property.json).
  5. Update the configuration by importing the JSON file. 
    clustermanager import config --componentId c1057ef0-ab4e-459f-8413-0e5036c65b7c --componentType logservice --file /home/builder/property.json
Updating the TMGC after the change
Successfully updated the TMGC component c1057ef0-ab4e-459f-8413-0e5036c65b7c for components of type logservice
Successfully imported configuration properties for the scope map[cluster:tml52clusterpune1 component:c1057ef0-ab4e-459f-8413-0e5036c65b7c zone:eastus] for components of type logservice

    For more information, see JSON Configuration Files Format for Different Outputs.