File Collection Parallelism - Overview

File parallelism allows you to configure concurrent file collections at the rule level in the file collector.

By default, LogLogic LMI processes a single file pull rule at a time by delivering files associated with a single rule. Where you feel appropriate and where the file transfer is relatively slow, LogLogic LMI allows for simultaneous file delivery for several rules simultaneously. This creates a higher throughput of file log processing because LogLogic LMI is capable of processing files at a relatively high speed compared to the time it takes to pull the files into LogLogic LMI. You can experiment by increasing the parallelism until there is a backlog in the /loglogic/data/filecollector/archiver directory.

File Parallelism is configured through command-line options that pass to the file collector when it is launched. These options are specified in /loglogic/conf/node_config.xml.

The number of concurrent file collections is specified by the command-line option -p x (where x is the degree of parallelism). The maximum is 100, and the default is 1 (when the command-line option is absent).
Note: File collection parallelism is available in LogLogic LMI 5.6.2 and later.