Remote Files

LogLogic® Universal Collector can collect files remotely and forward them to LogLogic LMI.

By default, LogLogic® Universal Collector pulls logs every 1 hour, but it can also pull every X minutes, after every X hours, daily at X time, or weekly on Y day at X time. LogLogic® Universal Collector supports FTP, SFTP, CIFS, and File protocol for remote file collection. In LogLogic® Universal Collector 2.9.0 SMB 2.1 support is added for CIFS protocol. Ensure that you use forward slash in the file path for CIFS and File protocol.

Note: To ensure there are no performance issues, use a physical machine for remote file collection. Do not use the remote file collection feature of LogLogic® Universal Collector to collect large remote files (above 1GB) on Virtual Machine systems which slows down the system significantly.

Remote File with Rotation

In the case of log file rotation, a log file is retired and renamed to a “rotated” name, and the monitored file is replaced by a new log file. Therefore, periodically during the monitoring of a log file that is rotated, the file is replaced by a new log file.

When the date field is checked for rotation, LogLogic® Universal Collector will only collect files that are modified after the remote file log source creation time.

LogLogic® Universal Collector deals with file rotation in two different ways. For more information, refer to Log File Rotation.

Note: LogLogic® Universal Collector supports only .GZ compression format for file rotation.

Remote File with No Rotation

  • Common Files Collection

    Ensure that you specify the correct file path on the remote file system to pull the file correctly.

    Note: You must use only the supported Windows characters to specify the file path or file name.
  • Compressed Files Collection

    LogLogic® Universal Collector collects the compressed file in four formats: .zip, .tgz, .tar.gz and .gz. The compressed file can not include compressed directories.

    The type of compression depends upon the compressed file extension.

  • Directory contains plain text log files

    Directory pull allows you to choose a directory and pull files from that directory based on the ‘include’ or ‘exclude’ options provided to you. Directory pull does not support file rotation. Directory contains the files in compressed format.

    Example: /loglogic/ directory has three files: a.txt, b.txt, c.txt

    Scenario 1: if users enters * for include, it will pull a.txt, b.txt, c.txt

    Scenario 2: if users enters *.txt for include and a.txt for exclude, it will pull b.txt and c.txt

    Scenario 3: if users enters a.txt for include and nothing for exclude, it will only pull a.txt.

  • Directory contains compressed files: LogLogic® Universal Collector supports collection of four compressed file formats including: .zip, .tgz, .tar.gz and .gz.
  • Using wildcard for directory pull

    Example: There are the following files:

    //sharepoint/mainFolder/subFolder1/*.log

    //sharepoint/mainFolder/subFolder2/*.log

    //sharepoint/mainFolder/subFolder3/*.log

    ...

    //sharepoint/mainFolder/subFolder9/*.log

    Scenario : if users enters //sharepoint/mainFolder/subFolder*/ or //sharepoint/mainFolder/subFolder?/ or //sharepoint/mainFolder/*/ for directory path and *.log for include, it will pull all the log files.

    Note: Do not use multiple wildcards in a directory path for directory pull.
    For Example:
    • Incorrect: //sharepoint/*/subfolder1/*.log
    • Correct: //sharepoint/mainFolder/subFolder1/*.log

Remote File Collection with SFTP protocol

LogLogic® Universal Collector Remote File Collection supports SFTP server by using password and SSH public key authentication.

  1. Open the LogLogic® Universal Collector Console.
  2. On the Collection tab, click New > Remote File.
  3. Enter the relevant information on the Log Source Edition screen and select SFTP protocol in the Collection pane.
  4. Select Public Key option, then enter the user Id and click Browse to upload the private key file.

    LogLogic® Universal Collector supports only DSA as signature algorithms for key pairs.

    Note: Select No passphrase while generating key pairs.