TML Consumer

Access Log Sync

The Access Log Sync process is deployed as standalone running java process inside the Log-service container. Its primary job is to sync the generated access logs to MOM environment via HTTP call.

Here are some of the prerequisites for access logs to be synced to MOM SaaS environment:

MOM host specified should be accessible via direct HTTPS connection from Mashery Local cluster.
The properties apiKey and apiSecret are required for MOM connection need to be passed through tml_log_properties.json as described in the section Properties for access log sync to MOM (tethered mode) in Exporting Access Logs.

Process location (tml-log container)

/opt/mashery/access-log-sync/

/bin - Startup script for access-log-sync process, for example: startup_accesslog_sync.sh
/conf - Process configuration, for example: logsync.properties
/lib - Library jars location
/lib/tibco - TIBCO internal libraries
/lib//tpcl. - External libraries

Process startup

The Access Log Sync process is a part of the Log-service container, and its lifecycle is managed by the log service.

Startup:

Start up of the Log Sync process is managed by the parent script /opt/mashery/containeragent/bin/startup_logservice.sh script.
This script is called by containeragent itself on container setup and it starts access-log-sync after it starts td-agent process.
Lifecycle options such as start/stop/restart/status of log service governs the same for both process (in the following order):
- startup_td-agent.sh
- startup_accesslog_sync.sh

Deployment Mode

Tethered mode (MOM Sync is enabled): In this mode, log-sync will be running as a java process, and it will perform the following:
- Identify logs that need to be sent - access logs will continue getting written in default location, for example, /mnt/data/trafficmanager/access/access_worker<n>.log. Currently the access logs will be rotated based on configuration parameter out_mlfileout.rb: log_max_size (default value : 1MB).
  1. After every write of log_max_size, existing access.log file will be rotated to access.log.1, access.log.2 and so on.
  2. Once total size of rotated log files exceeds sync data size limit (default ~10mb), it will trigger Log sync job.
  3. Once Log sync job is triggered, it will move all rotated log files to /mnt/data/trafficmanager/queue/ folder. After moving, compress all these files to zip having name in format log_epochFrom_log_epochTo_machineInstanceName.zip.
  4. Apart from above size based trigger, log sync job will continue to run on every configured sync time interval defined in logsync.properties (default: 15 min).
    
    Note: In the scheduler sync, if access log files are not rotated (doesn't exceed log_max_size : 1MB), the access log accumulated so far will still be synced with the name of access.log.0 and moved to archive directory.
  5. The total size of log files compressed to be sent should not exceed 10 MB (configurable). After this limit, zip will not include any further log files for sync call.
- Data enrichment on log-sync.
  1. Read logs from queue folder.
  2. Send ONLY access logs from Traffic Manager containers - No need for cleanup and de-duplication.
  3. Enrichment of data_source_id and capacity_center_id/cluster_id - log-sync process will call MOM endpoint using credentials and fetch MOM metadata to enrich logs (similar to Mashery Local 4.4.0).
- Archive already sent data.
  1. Once sent, move logs from /mnt/data/trafficmanager/queue/ to /mnt/data/trafficmanager/archive/timestamp/
- Maintain audit of sync done.
  1. Write all the metrics to a file in /mnt/data/tmgc-log/{LOG_HOST_NAME}/accesslogconsumer/syncaudit/syncaudit.log in the form of key-value pair.
- Status of the sync process (running, stopped, etc.)
Untethered mode (MOM Sync is disabled): In this mode, the access-log-sync though is configured but will not run. A check is put in the startup script to check if MOM Credentials is configured in /opt/mashery/access-log-sync/conf/logsync.properties. If not, it will not start.

Sync Interval change by client

You can change the sync interval in minutes [range 1-59] in the scope of a cluster/zone using CLI. For example:

Syntax: clustermanager set lsi --componentId <log_service_component_id> --componentType logservice --syncInterval 12
 
eg :
root@cm-deploy-0-777dd4ff67-vlfjz builder]# clustermanager set lsi --componentId 9d7e4de0-c11e-42b4-86b8-4295ff4286df --componentType logservice --syncInterval 12
 
Calling POST url: http://xxx.xx.x.x:9080/log/loggers/access/sync/configuration?syncinterval=12
Successfully updated log sync interval 12 for log component 9d7e4de0-c11e-42b4-86b8-4295ff4286df
[root@cm-deploy-0-777dd4ff67-vlfjz builder]#

This new sync Interval will be replaced in /opt/mashery/access-log-sync/conf/logsync.properties and will be readily picked up by LogSyncJob to start syncing according to this interval.

Appender and logger is added in containeragent logback.xml using Syslogappender.
Process logs are stored in the location: /mnt/data/tmgc-log/{LOG_HOST_NAME}/accesslogconsumer/consumer.log (directory name will be governed by metadata defined in logback.xml).
Audit logs comprising of log sync result will be written to location: /mnt/data/tmgc-log/{LOG_HOST_NAME}/accesslogconsumer/syncaudit/syncaudit.log

Contents