Configuring Parallel File Processing and Parallel File Forwarding

This section describes how to enable and configure the number of parallel file processing and file forwarding threads in the file collector and event forwarding process.

By default, a single process handles a pulled log file on LogLogic LMI. For appliances where multiple files are collected simultaneously, LogLogic LMI allows for parallel file processing, that is, parsing, time extraction, and log conversion. This allows for faster file handling. LogLogic LMI provides the flexibility to adjust overall file handling by selecting an appropriate degree of parallelism. As a rule of thumb, the number of parallel processes should be targeted at the number of cores minus 1. If the number is too high, then processing scalability is not linear. LogLogic LMI selects its processing rate automatically.

It is good practice to perform parallel file processing when a backlog of unprocessed files is accumulated in the /loglogic/data/filecollector/forwarder directory. If a backlog is detected in this directory, the user can increase file forwarding throughput by making it parallel as well.

Procedure

  1. Create the file /loglogic/conf/fc.conf and specify the NumParallelFwdProcs and NumParallelParserProcs parameters in the file. For example: 
    NumParallelFwdProcs=3
NumParallelParserProcs=10
  2. Run the command: 
    $mtask -s engine_filecollector restart
    Keyword Description
    NumParallelFwdProcs Indicates the number of processes that forward files in parallel.

    The maximum value of this parameter is the number of cores on LogLogic LMI.

    Example: NumParallelFwdProcs=3

    NumParallelParserProcs Indicates the number of processes for parallel file processing.

    The maximum value for this parameter is the number of cores on LogLogic LMI.

    Example: NumParallelParserProcs=10