Definition of Configuration Rule Files
You must define a configuration file containing a rule to format log messages before they are forwarded.
The formatted forwarding feature is used to format log messages prior to forwarding. The Forwarding function uses the configuration file that defines the formatting rules. All log messages that match the forwarding rule are formatted.
Rules consist of Regular expressions that will be applied to the log messages. If log messages are matched with the Regular expression, then the extracted strings will be substituted into the forwarded message before forwarding it to the defined destination.
You can upload only one configuration file for each Message Routing rule. Each configuration rule file can be used across multiple destinations or rules, and can have a maximum file size of 1 GB.
The configuration rule file is used for:
- filtering logs
- transforming/formatting log messages
- filtering character patterns from a log (shredding) and, optionally, replacing them with literal strings
A configuration rule file can consist of multiple rules. A rule consist of either regexp/template pattern or shred/replace pattern.
A regexp/template rule consists of two lines; a Regular expression used to match and extract patterns and the forwarded message template. A shred/replace pattern rule defines shred= option. You can also have both types of rules in the same configuration rule file.
First, regexp/template rules are applied in the order of their appearance in the configuration file. The system will stop applying rules as soon as the first match is detected. The shred/replace rules are applied after a single scan across all regexp/template rules. Only those log messages that match any regexp rule will be forwarded. However, if you use forward all option, any log message will be forwarded even without matching any regexp rule.
The configuration rule file must satisfy the following criteria:
- rule file cannot be empty
- must consist of one or more rules; either regexp/template rule or shred/replace rule or both types can be in the same rule file
- regexp <regular expression> must be valid; cannot be empty
regexp is a keyword followed by a PCRE-compatible regular expression. A regexp has to succeed in order for a log message to be modified.
For example,
regexp(.+)\s(.+)\slx_scheduler:\s(.+)\s(.+):\s\((.+)\)\slx_scheduler:\sending\sconfig\smgmt\sjob
- template must be valid; cannot be empty
The rule has to specify a template for the contents of the forwarded log. The template can contain literal strings as well as portions of the original message. The template is applied to create a new message. After template-driven formatting is completed, the new log message will be forwarded to the destination of the forwarding rule. $1, $2, and so on, in the template refer to the patterns extracted by regexp. Extracted patterns are defined by a matching pair of parentheses, for example (.+) in the previous regular expression example. $1 refers to the first pair of parentheses.
It is good practice to use up to $10 matching patterns in the template. You can use the matching patterns repeatedly, however, the digits after the $ sign should be 10 or less than 10.
For example,
template LOGS|LogLogic MODEL|CC01|Application configuration change|5|deviceExternalId=62968-1 msg=ending config mgmt job shost=$2 dhost=$2 suser=system suid=system spriv=User
- shred/replace option (Optional)
Shred option removes the portion of the transformed message based on the template and replaces with the new string, and then forwards the message to the destination. For example,
shred=\d{16} replace=XYZ-XYZ-XYZ-XYZ
shred= regular expression is applied to every message repeatedly in order to identify a sensitive pattern. The sensitive pattern can have more than one instance in the message, all instances will be detected. The shred in the config file must be all in lower-case letters.
replace= optionally defines a substitution string that replaces sensitive string.