Log Service

What's Inside

td-agent/fluentd

Services

Log Producer Service
Log Consumer Service

Resource Dependencies

Log Consumers
Registry Binding

Configuration Properties

Name	Default Value	Comments
api_key		Only needed for "Tethered" setup.
api_secret		Only needed for "Tethered" setup.
td_agent_output_channelType	DEFAULT (always enabled)	Set this property if you need to send access logs to other destinations. Accepted values are DEFAULT, FORWARD, ELASTICSEARCH, KAFKA, HTTP and TCP. Changing to anything other than DEFAULT requires additional properties: Property Name / Default Value td_agent_out_forward_host / NULL td_agent_out_elasticsearch_host / NULL td_agent_out_elasticsearch_port / NULL td_agent_out_elasticsearch_index / $ {TAG} td_agent_out_http_URI / NULL td_agent_out_kafka_brokers / NULL td_agent_out_kafka_topic / ${TAG} td_agent_out_tcp_host / NULL td_agent_out_tcp_port / NULL
mom_host	https://api-mom.mashery.com	Only needed for "Tethered" setup.
td_agent_bit_buffer_type	filesystem	The back pressure buffer type for logs - store in memory or back up to disk. In most conditions, it would not be required to change the default value; but if high disk i/o is noticed, change the setting. Valid values are `memory` and `filesystem`.
td_agent_bit_metrics_buffer_type	filesystem	The back pressure buffer type for metrics - store in memory or back up to disk. In most conditions, it would not be required to change the default value; but if high disk i/o is noticed, change the setting. Valid values are `memory` and `filesystem`.

Note: td_agent_bit_buffer_type and td_agent_bit_metrics_buffer_type properties can be set at deployment time or can be modified using the Cluster Manager CLI.

Example:

cm ls components

clustermanager set config --componentType <component type from ls components> --componentId <component id from ls components> --prop td_agent_bit_buffer_type --value filesystem

TD agent workers configuration

The performance of the log service is dependent on a combination of amount of traffic volume as follows; velocity of access logs, and a number of workers within the log service handling the traffic. A simple thumb rule will be - more traffic means more workers. Thus, in an expectation of amount of traffic; the log pod will dictate necessary compute, memory, and storage. The workers are distributed in sequence groups to different types of logs. An error in these values can cause the log service not to function as expected. Workers are identified by a number that starts with zero and ends with 1 index less than the number of workers. For example, when number of workers is mentioned as 10; then the workers' index starts with 0 and ends at 9. An ideal distribution of workers will ensure that access and payload(if enabled) logs get more workers than container and metrics. Each worker denotes a separate process handling a specific type of logs.

Number of workers for access logs and payloads should be same, because both work of traffic would be handling similar velocity of logs.

Key	Description	Expected Values	Default value	Example
td_agent_accesslog_workers	A set of workers assigned for processing access logs. Higher the traffic, more is the number of workers required.	Range of positive value represented as a String.	5-9	td_agent_accesslog_workers="6-10"
td_agent_containerlog_workers	A set of workers assigned for processing logs of different services.	Range of positive value represented as a String.	3-4	td_agent_containerlog_workers="3-6"
td_agent_metrics_workers	A set of workers assigned for processing metrics of service running on all pods.	Range of positive value represented as a String	1-2	td_agent_metrics_workers="1-4"
td_agent_payload_logging_workers	A set of workers assigned for processing payload logs. Higher the traffic, more is the number of workers required.	Range of positive value represented as a String	10-14	td_agent_payload_logging_workers="15-19"
td_agent_dimdata_workers	A worker assigned for processing metadata for access logs. Recommendation: Do not assign any other worker or use 0 for any other logs.	Positive single value	0	td_agent_dimdata_workers="1"
td_agent_workers	Total number of workers to be made available	Positive integer	10	td_agent_workers="20"

Contents