TIBCO Cloud™ API Management - Local Edition Access Log Sync

The Access Log Sync process is deployed as a standalone running a Java process inside the Log-service container. Its primary job is to sync the generated access logs to the MOM environment via an HTTP call.

The access log sync is applicable only for tethered mode setup.

Prerequisites

MOM host specified should be accessible via direct HTTPS connection from TIBCO Cloud™ API Management - Local Edition cluster.
The properties apiKey and apiSecret are required for MOM connection need to be passed through tml_log_properties.json as described in the section Properties for access log sync to MOM (tethered mode) in Exporting Access Logs.

Process location (tml-log container)

/opt/mashery/access-log-sync/

/bin - Startup script for access-log-sync process, for example: startup_accesslog_sync.sh
/conf - Process configuration, for example: logsync.properties
/lib - Library jars location
/lib/tibco - TIBCO internal libraries
/lib//tpcl. - External libraries

Process startup

The Access Log Sync process is a part of the Log-service container, and its lifecycle is managed by the log service.

Startup:

Start up of the Log Sync process is managed by the parent script /opt/mashery/containeragent/bin/startup_logservice.sh script.
This script is called by containeragent itself on container setup and it starts access-log-sync after it starts td-agent process.
Lifecycle options such as start/stop/restart/status of log service governs the same for both process (in the following order):
- startup_td-agent.sh
- startup_accesslog_sync.sh

Deployment Mode

Tethered mode (MOM Sync is enabled): In this mode, log-sync will be running as a Java process, and it will perform the following:
- Identify logs that need to be sent - access logs will continue getting written in default location, for example, /mnt/data/trafficmanager/access/access_worker<n>.log. Currently the access log rotates based on the configuration parameter out_mlfileout.rb: log_max_size. The default value is 1MB.
  1. After every write of log_max_size, the existing access.log file will be rotated to access.log.1, access.log.2 and so on.
  2. If the total size of the rotated log files exceeds the sync data size limit (default ~10 MB), it triggers the Log sync job.
  3. After the Log sync job is triggered, it will move all rotated log files to the /mnt/data/trafficmanager/queue/ folder. After moving, these files are compressed to a .zip file with the name, log_epochFrom_log_epochTo_machineInstanceName.zip.
  4. Apart from the above size-based trigger, the Log sync job will continue to run on every configured sync time interval defined in logsync.properties (default: 15 min).
    
    Note: In the scheduler sync, if access log files are not rotated (doesn't exceed log_max_size : 1 MB), the access log accumulated so far will still be synced with the name of access.log.0 and moved to archive directory.
  5. The total size of the compressed log files should not exceed 10 MB (configurable). After this limit, the .zip file will not include any further log files for sync call.
- Data enrichment on log-sync.
  1. Read logs from queue folder.
  2. Send ONLY access logs from Traffic Manager containers - No need for cleanup and de-duplication.
  3. Enrichment of data_source_id and capacity_center_id/cluster_id - log-sync process will call MOM endpoint using credentials and fetch MOM metadata to enrich logs.
- Archive already sent data.
  1. Once sent, move logs from /mnt/data/trafficmanager/queue/ to /mnt/data/trafficmanager/archive/timestamp/
- Maintain audit of sync done.
  1. Write all the metrics to a file in /mnt/data/tmgc-log/{LOG_HOST_NAME}/accesslogconsumer/syncaudit/syncaudit.log in the form of key-value pair.
- Status of the sync process (running, stopped, etc.)
Untethered mode (MOM Sync is disabled): In this mode, the access-log-sync though is configured but will not run. A check is put in the startup script to check if MOM Credentials are configured in /opt/mashery/access-log-sync/conf/logsync.properties. If not, it will not start.

Sync Interval change by client

You can change the sync interval in minutes [range 1-59] in the scope of a cluster/zone using CLI. For example:

Syntax: clustermanager set lsi --componentId <log_service_component_id> --componentType logservice --syncInterval 12
 
eg :
root@cm-deploy-0-777dd4ff67-vlfjz builder]# clustermanager set lsi --componentId 9d7e4de0-c11e-42b4-86b8-4295ff4286df --componentType logservice --syncInterval 12
 
Calling POST url: http://xxx.xx.x.x:9080/log/loggers/access/sync/configuration?syncinterval=12
Successfully updated log sync interval 12 for log component 9d7e4de0-c11e-42b4-86b8-4295ff4286df
[root@cm-deploy-0-777dd4ff67-vlfjz builder]#

This new sync Interval will be replaced in /opt/mashery/access-log-sync/conf/logsync.properties and will be readily picked up by LogSyncJob to start syncing according to this interval.

Appender and logger is added in container agent logback.xml using Syslogappender.
Process logs are stored in the folder, /mnt/data/tmgc-log/{LOG_HOST_NAME}/accesslogconsumer/consumer.log (directory name will be governed by metadata defined in logback.xml).
Audit logs comprising of log sync result will be written to folder,/mnt/data/tmgc-log/{LOG_HOST_NAME}/accesslogconsumer/syncaudit/syncaudit.log

Contents