Changing Output Destinations for Different Types of Logs using CLI

This topic contains information about how to configure output channels to send logs, access logs and metrics to Kafka, Elasticsearch, Syslog, Generic TCP, or HTTP end points.

How to Use This Feature

This feature allows you to change the output destination for different types of logs in one invocation.

Caution: Using the CLI will overwrite the existing configuration. Therefore, each time it is used the system expects to receive a full configuration. Incomplete configurations are ignored when generating the log service's configurations.

The command accepts a JSON that contains the different configuration properties. The data in the file should adhere to JSON rules. All properties are top-level attributes. See the examples below.

Command Line

The generic form of the command is: clustermanager --componentType logservice [componentid] --file <full path to json>

Example: 

cm 
import
 config [--componentId 3a076250-955b-49ef-ab06-ea19df10a665 --componentType logservice --file props.json

Types of Logs

You can export the following three types of logs to the chosen destination:
  1. Access logs
  2. Metrics
  3. Container logs

Each type of log has a different set of property names. This way you can send different types of logs to different destinations.

The logs are written to the disk (DEFAULT) on the log service container irrespective of the destination you choose.

All values and property names are in character/String data type. Depending on chosen output channel type, the remaining configuration will change.

Access Logs
Destination Type Name in JSON Description Notes Example
td_agent_output_channelType Comma separated string of types of channels Applicable values are ELASTICSEARCH, HTTP, KAFKA, SYSLOG,TCP "td_agent_output_channelType": "DEFAULT,ELASTICSEARCH"
ELASTICSEARCH td_agent_out_elasticsearch_host Host name or IP address of Elastic search server or ELB The hostname should be reachable from within the log service container "td_agent_out_elasticsearch_host" : "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_elasticsearch_port "td_agent_out_elasticsearch_port" : "9200"
td_agent_out_elasticsearch_index "td_agent_out_elasticsearch_index" : "ml5_access_logs"
HTTP td_agent_out_http_URI HTTP URI to which logs are to be sent The HTTP URI should be accessible from the log container
KAFKA td_agent_out_kafka_brokers Comma delimited list of kafkabroker_host:port The Kafka brokers host should be reachable from the log service and broker should be running "td_agent_out_kafka_brokers":"broker1:9092,broker2:9092"
td_agent_out_kafka_topic "td_agent_out_kafka_topic" : "ml5_access_logs"
SysLog td_agent_out_syslog_host Host name of IP address where syslog is running Host should be reachable from log service "td_agent_out_syslog_host": "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_syslog_port Port number of syslog program "td_agent_out_syslog_port":"5142"
td_agent_out_syslog_tag Tag with which to uniquely identify the log defaults to incoming log event tag "td_agent_out_syslog_tag":"ml5_access_logs"
td_agent_out_syslog_protocol Protocol on which syslog is listening for incoming requests defaults to tcp "td_agent_out_syslog_protocol":"tcp"
TCP td_agent_out_tcp_host Host name where a TCP socket is listening Host should be reachable from log service "td_agent_out_tcp_host": "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_tcp_port Port on which the TCP socket is listening "td_agent_out_tcp_port":"6000"
Metrics
Destination Type Name in JSON Description Notes Example
td_agent_metric_output_channelType Comma separated string of types of channels Applicable values are ELASTICSEARCH, HTTP, KAFKA, SYSLOG,TCP "td_agent_metric_output_channelType": "DEFAULT,ELASTICSEARCH"
ELASTICSEARCH td_agent_out_metric_elasticsearch_host Host name or IP address of Elastic search server or ELB The hostname should be reachable from within the log service container "td_agent_out_metric_elasticsearch_host" : "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_metric_elasticsearch_port "td_agent_out_metric_elasticsearch_port" : "9200"
td_agent_out_metric_elasticsearch_index defaults to incoming log event tag "td_agent_out_metric_elasticsearch_index" : "ml5_metric_logs"
HTTP td_agent_out_metric_http_URI HTTP URI to which logs are to be sent The HTTP URI should be accessible from the log container "td_agent_out_metric_http_URI": "http://a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com /ml5/metriclogs"
KAFKA td_agent_out_metric_kafka_brokers Comma separated list of kafkabroker_host:port The Kafka brokers host should be reachable from the log service and broker should be running "td_agent_out_metric_kafka_brokers":"broker1:9092,broker2:9092"
td_agent_out_metric_kafka_topic "td_agent_out_metric_kafka_topic" : "ml5_metric_logs"
SysLog td_agent_out_metric_syslog_host Host name of IP address where syslog is running Host should be reachable from log service "td_agent_out_metric_syslog_host": "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_metric_syslog_port Port number of syslog program "td_agent_out_metric_syslog_port":"5142"
td_agent_out_metric_syslog_tag Tag with which to uniquely identify the log defaults to incoming log event tag "td_agent_out_metric_syslog_tag":"ml5_metric_logs"
td_agent_out_metric_syslog_protocol Protocol on which syslog is listening for incoming requests Defaults to tcp "td_agent_out_metric_syslog_protocol":"tcp"
TCP td_agent_out_metric_tcp_host Host name where a TCP socket is listening Host should be reachable from log service "td_agent_out_metric_tcp_host": "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_metric_tcp_port Port on which the TCP socket is listening "td_agent_out_metric_tcp_port":"6000"
Container Logs
Destination Type Name in JSON Description Notes Example
td_agent_container_output_channelType Comma separated string of types of channels Applicable ones are ELASTICSEARCH, HTTP, KAFKA, SYSLOG,TCP "td_agent_container_output_channelType": "DEFAULT,ELASTICSEARCH"
ELASTICSEARCH td_agent_out_container_elasticsearch_host The hostname should be reachable from within the log service container "td_agent_out_container_elasticsearch_host" : "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_container_elasticsearch_port "td_agent_out_container_elasticsearch_port" : "9200"
td_agent_out_container_elasticsearch_index defaults to incoming log event tag "td_agent_out_container_elasticsearch_index" : "ml5_container_logs"
HTTP td_agent_out_container_http_URI HTTP URI to which logs are to be sent The HTTP URI should be accessible from the log container "td_agent_out_container_http_URI": "http:// a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com /ml5/containerlogs"
KAFKA td_agent_out_container_kafka_brokers Comma separated list of kafkabroker_host:port The Kafka brokers host should be reachable from the log service and broker should be running "td_agent_out_container_kafka_brokers":"broker1:9092,broker2:9092"
td_agent_out_container_kafka_topic Defaults to incoming log event tag "td_agent_out_container_kafka_topic" : "ml5_container_logs"
Syslog td_agent_out_container_syslog_host Host name of IP address where syslog is running Host should be reachable from log service "td_agent_out_container_syslog_host": "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_container_syslog_port Port number of syslog program "td_agent_out_container_syslog_port":"5142"
td_agent_out_container_syslog_tag Tag with which to uniquely identify the log Defaults to incoming log event tag "td_agent_out_container_syslog_tag":"ml5_containerlogs"
td_agent_out_container_syslog_protocol Protocol on which syslog is listening for incoming requests Defaults to TCP "td_agent_out_container_syslog_protocol":"tcp"
TCP td_agent_out_container_tcp_host Host name where a TCP socket is listening Host should be reachable from log service "td_agent_out_container_tcp_host": "a06ebe8a15a3011e9bf380608b2b6e73-531419666.us-east-1.elb.amazonaws.com"
td_agent_out_container_tcp_port Port on which the TCP socket is listening "td_agent_out_container_tcp_port":"6000"

Steps to Import Configs From Clustermanger Command Line Interface

  1. List components 
    clustermanager ls components
Using cluster [tml52clusterpune1]
Using Zone [eastus]
Component ID                          Component Type       Component Name       Component Status     Component Host       Component Agent Port   Component Service Port(s) 
------------------------------------- ------------------- -------------------- -------------------- -------------------- ---------------------- ---------------------------
f41c95b9-6059-42c1-8a4f-fb2261974c14  cache                cache-set-0-0.cac... ACTIVE               10.244.1.6           9080                   11212,11211,11213,11214,11215,11216
710b9df4-399a-4551-b1ed-c1fc1f8ed744  configmanager        cm-deploy-0-85fbf... ACTIVE               10.244.2.6           9080                   8080                      
c1057ef0-ab4e-459f-8413-0e5036c65b7c  logservice           log-set-0-0.log-s... ACTIVE               10.244.2.7           9080                   24224                     
d516fe9e-3ac3-4a09-a9b0-1dceba36a23d  nosql                cass-set-0-2.cass... ACTIVE               10.244.1.5           9080                   9042                      
29812c89-2e87-4dc0-a981-890cd1a72933  nosql                cass-set-0-0.cass... ACTIVE               10.244.2.5           9080                   9042                      
99950720-6ded-46be-a87f-dc82be47f265  nosql                cass-set-0-1.cass... ACTIVE               10.244.0.6           9080                   9042                      
8a5a78f0-f032-4046-ba59-28a75004c9ce  sql                  mysql-set-0-0.mys... ACTIVE               10.244.2.8           9080                   3306                      
111628dc-7893-45c8-8add-8a7944a2ff8c  trafficmanager       tm-deploy-0-5f787... ACTIVE               10.244.0.7           9080                   8080
  2. Get the component ID of logservice from the output.
  3. Check the existing configuration. 
    clustermanager ls config --componentId c1057ef0-ab4e-459f-8413-0e5036c65b7c --componentType logservice
clusterId [ad41097e-35bc-4fc4-bb22-084dc25fdbf2] and zoneId [9046178a-00f9-4d59-a92e-59824142df56]
No settings found for the given componentId [c1057ef0-ab4e-459f-8413-0e5036c65b7c] of type [logservice]
  4. Create a JSON file for input (e.g.: /home/builder/property.json).
  5. Update the configuration by importing the JSON. 
    clustermanager import config --componentId c1057ef0-ab4e-459f-8413-0e5036c65b7c --componentType logservice --file /home/builder/property.json
Updating the TMGC after the change
Successfully updated the TMGC component c1057ef0-ab4e-459f-8413-0e5036c65b7c for components of type logservice
Successfully imported configuration properties for the scope map[cluster:tml52clusterpune1 component:c1057ef0-ab4e-459f-8413-0e5036c65b7c zone:eastus] for components of type logservice

JSON Config files format for Different Outputs

Example JSON for ELASTICSEARCH output. 

{
    "td_agent_output_channelType": "DEFAULT,ELASTICSEARCH",
    "td_agent_container_output_channelType": "DEFAULT,ELASTICSEARCH",
    "td_agent_metric_output_channelType": "DEFAULT,ELASTICSEARCH",
    "td_agent_out_elasticsearch_host" : "<Elasticsearch Host IP>",
    "td_agent_out_elasticsearch_port" : "9200",
    "td_agent_out_elasticsearch_index" : "ml52_access",
    "td_agent_out_container_elasticsearch_host" : "<Elasticsearch Host IP>",
    "td_agent_out_container_elasticsearch_port" : "9200",
    "td_agent_out_container_elasticsearch_index" : "tml52_container",
    "td_agent_out_metric_elasticsearch_host" : "<Elasticsearch Host IP>",
    "td_agent_out_metric_elasticsearch_port" : "9200",
    "td_agent_out_metric_elasticsearch_index" : "ml52_metric"  
}

Example JSON for KAFKA output. 

{
    "td_agent_output_channelType": "DEFAULT,KAFKA",
    "td_agent_container_output_channelType": "DEFAULT,KAFKA",
    "td_agent_metric_output_channelType": "DEFAULT,KAFKA",
    "td_agent_out_kafka_brokers" : "<Kafka Host IP>:9092",
    "td_agent_out_kafka_topic" : "tml52_access_logs",
    "td_agent_out_metric_kafka_brokers" : "<Kafka Host IP>:9092",
    "td_agent_out_metric_kafka_topic" : "tml52_metric_logs",
    "td_agent_out_container_kafka_brokers" : "<Kafka Host IP>:9092",
    "td_agent_out_container_kafka_topic" : "tml52_container_logs"   
}

Example JSON for HTTP output. 

{
    "td_agent_output_channelType": "DEFAULT,HTTP",
    "td_agent_container_output_channelType": "DEFAULT,HTTP",
    "td_agent_metric_output_channelType": "DEFAULT,HTTP",
    "td_agent_out_http_URI":"<HTTP URI>",
    "td_agent_out_metric_http_URI":"<HTTP URI>",
    "td_agent_out_container_http_URI":"<HTTP URI>"   
}

Example JSON for TCP output. 

{
    "td_agent_output_channelType": "DEFAULT,TCP",
    "td_agent_container_output_channelType": "DEFAULT,TCP",
    "td_agent_metric_output_channelType": "DEFAULT,TCP",
    "td_agent_out_tcp_host":"<TCP HOST>",
    "td_agent_out_tcp_port":"6000",
    "td_agent_out_metric_tcp_host":"<TCP HOST>",
    "td_agent_out_metric_tcp_port":"6000",
    "td_agent_out_container_tcp_host":"<TCP HOST>",
    "td_agent_out_container_tcp_port":"6000" 
}

Example JSON for SYSLOG output. 

{
    "td_agent_output_channelType": "DEFAULT,SYSLOG",
    "td_agent_container_output_channelType": "DEFAULT,SYSLOG",
    "td_agent_metric_output_channelType": "DEFAULT,SYSLOG",
    "td_agent_out_syslog_host":"<SYSLOG HOST>",
    "td_agent_out_syslog_port":"5142",
    "td_agent_out_syslog_tag":"ml5_accesslogs",
    "td_agent_out_syslog_protocol":"tcp",
    "td_agent_out_metric_syslog_host":"<SYSLOG HOST>",
    "td_agent_out_metric_syslog_port":"5142",
    "td_agent_out_metric_syslog_tag":"ml5_metriclogs",
    "td_agent_out_metric_syslog_protocol":"tcp",
    "td_agent_out_container_syslog_host":"<SYSLOG Host>",
    "td_agent_out_container_syslog_port":"5142",
    "td_agent_out_container_syslog_tag":"ml5_containerlogs",
    "td_agent_out_container_syslog_protocol":"tcp"
}

Example JSON file with multiple output. 

{
    "td_agent_output_channelType": "DEFAULT,ELASTICSEARCH,KAFKA",
    "td_agent_container_output_channelType": "DEFAULT,ELASTICSEARCH,KAFKA",
    "td_agent_metric_output_channelType": "DEFAULT,ELASTICSEARCH,KAFKA",
    "td_agent_out_elasticsearch_host" : "<Elasticsearch Host IP>",
    "td_agent_out_elasticsearch_port" : "9200",
    "td_agent_out_elasticsearch_index" : "ml52_access",
    "td_agent_out_container_elasticsearch_host" : "<Elasticsearch Host IP>",
    "td_agent_out_container_elasticsearch_port" : "9200",
    "td_agent_out_container_elasticsearch_index" : "tml52_container",
    "td_agent_out_metric_elasticsearch_host" : "<Elasticsearch Host IP>",
    "td_agent_out_metric_elasticsearch_port" : "9200",
    "td_agent_out_metric_elasticsearch_index" : "ml52_metric",
    "td_agent_out_kafka_brokers" : "<Kafka Host IP>:9092",
    "td_agent_out_kafka_topic" : "tml52_access_logs",
    "td_agent_out_container_kafka_brokers" : "<Kafka Host IP>:9092",
    "td_agent_out_container_kafka_topic" : "tml52_container_logs",
    "td_agent_out_metric_kafka_brokers" : "<Kafka Host IP>:9092",
    "td_agent_out_metric_kafka_topic" : "tml52_metric_logs"   
}