Adding File Transfer Rules
You can add and modify the existing File transfer rules from the Management > Devices> File Transfer Rules tab. The options on both tabs are the same.
Procedure
- If the new file transfer rule uses the SFTP or SCP protocol, perform a keycopy to the specified server for the specified user. See the system Command.
-
On the
Management
> Devices
> File Transfer Rules tab, select
Device Type and
Device, and then click
.
- In the Rule Name field, specify a name for the rule.
-
From the
Protocol drop-down list, select the protocol type to use to transfer files. The supported protocols include:
For details about protocols, see File Transfer Protocols.
- In the file transfer rules using the SCP protocol, if you specify a wildcard, make sure the list of files (list of file names only and not the file sizes), does not exceed 32KB. If the list of files exceeds 32KB, consider combining the files to reduce the size of the list to a single file or fewer files. Depending on the number of files you have, you may want to combine the files into an hourly, daily, or weekly file. For example, if you have a large number of .aud files in your specified file transfer directory, you can run the command tar jcf today.tar.bz2 *.aud to create a single file containing all of the .aud files in the directory. Ensure you configure your SCP file transfer rule to pick up the combined file; in this example today.tar.bz2.
- For file transfers using the HDFS protocol, LogLogic LMI connects to the default HDFS port 9000. Whether using the default or a custom port, ensure that the HDFS cluster is configured to use the same port that LogLogic LMI uses. If a different port is used, change the port in the --hdfs-port parameter in the configuration file /loglogic/conf/fc_hdfs.conf and then restart engine_filecollector.
- For the Amazon S3 protocol, you must adhere to the Amazon Bucket Restrictions and the rules for naming buckets. Ensure that correct retention rules are set up in LogLogic LMI for file transfer.
- In the User ID field, type the ID to use when accessing the file transfer log source.
- If you select FTP, FTPS, HDFS, HTTP, HTTPS, or CIFS in step 4 as a protocol, type the Password and verify the password associated with the User ID.
- If you select CIFS:
-
If you select Amazon S3, specify the AWS credentials in the
Access Key and
Secret Key fields.
-
In the
Files field, type the absolute path of the file you want to transfer. For example,
/root/user/LogLogic/IIS/ms_iis_ex030131.txt. Multiple files can be specified using a comma (,) or semicolon (;) as the delimiter for all protocols. For additional information, see the table in
File Transfer Protocols.
Note: The administrator of the remote system must verify that the file path does not include symbolic links. If you cannot specify a file path without symbolic links, then relocate the files to a path that can be specified without symbolic links.
- For file transfer using SFTP, files with .xml extension are not supported.
- For file transfer using FTP, FTPS, HTTP, and HTTPS, the file size limit of 20 GB is set in the corresponding configuration file. For example, for FTP files, the
/loglogic/conf/fc_ftp.conf file specifies the file size limit as:
--max-filesize 21474836480
-
For file transfer using Amazon S3, the file path must start with the bucket name. For example:
/<bucket-name>/<dirname>/<Log-file-name> or /<bucket-name>/<Log-file-name>
, where:
- From the File Format drop-down menu, select the format of the files to be transferred.
- Click the Test button to check if the connection parameters can be used to successfully retrieve the file without ingesting the data. The File Transfer Test Status window appears. Click Cancel to cancel the running test.
-
In the
Collection Time section, specify the time interval or schedule on which files should be transferred.
Option Description Every Select the number of minutes (in five minute increments) to wait between intervals. Every Select the number of hours to wait between intervals. Daily at Select the hour at which files should be transferred every day. Weekly on Select the day and time at which files should be transferred every week. -
For
Use Advanced Data Duplication Detection, click the
Yes radio button for the appliance to try to detect partial duplication of the newly transferred file, or
No for the Appliance not to detect partial duplication.
By default, the Appliance automatically detects exact duplicates among transferred files. Advanced detection analyzes potential duplicate data between transferred files, even if the files aren’t identical. (For example, a partial file at the end of a transfer that’s repeated in whole at the start of the next transfer.)
With or without Advanced Data Duplication Detection, the old file is always replaced with the newly transferred file. However, when duplication (complete or partial) is detected, only the non-duplicated portion of the file is processed.
- Click the Yes radio button to Enable the transfer rule or No to disable this rule.
-
Click
Add to add the rule.
Once you verify the rule displays in the Management > Devices > File Transfer Rules tab, you can click Dashboards > Log Source Status to view the transfer as it occurs. This might take several minutes.
- If the protocol you select from the Management > Devices > File Transfer Rules > Add Rule > Add File Transfer Rule tab requires a public key copy, register your public key on the server from which you transfer files.
Copyright © Cloud Software Group, Inc. All rights reserved.