Configuring Parallel File Processing and Parallel File-Forwarding

Using the following configuration, you can enable and configure the number of parallel file processing and file forwarding threads in the file collector and event forwarding process.

By default, a single process will handle a pulled log file on LogLogic LMI. For appliances where multiple files are collected simultaneously, LogLogic LMI allows for parallel file processing, that is, parsing, time extraction, and log conversion. This allows for faster file handling. LogLogic LMI provides the flexibility to adjust overall file handling by selecting an appropriate degree of parallelism. As a rule of thumb, the number of parallel processes should be targeted at the number of cores minus 1. If the number is too high, then processing scalability will not be linear. LogLogic LMI will select its processing rate automatically.

It is good practice to perform parallel file processing when a backlog of unprocessed files is accumulated in the /loglogic/data/filecollector/forwarder directory. If a backlog is detected in this directory, the user can increase file forwarding throughput by making it parallel as well.

Procedure

  1. In the backend of LogLogic LMI, run the command: mtask stop.
  2. Create the file /loglogic/conf/fc.conf and specify the NumParallelFwdProcs and NumParallelParserProcs parameters in the file. For example: 
    NumParallelFwdProcs=3
NumParallelParserProcs=10
  3. In the file /loglogic/conf/node_config.xml, specify the args parameter in the engine_filecollector section of corresponding node type. For example: 
    args="-p 4"
  4. Run the command: mtask start.