GP Parser-Based Data Models

The data models in LogLogic LMI can be broadly classified into the following categories:

• Advanced Data Models, which use parser types such as syslog, Regex, JSON, and so on.
• GP Parser-Based Data Models, which are data models that use a grouped-pattern parser (GP parser). GP parsers are especially efficient in handling complex Regex parsing rules that work on heterogeneous free-text logs.

GP parsers are very different in structure and syntax from the advanced data model parsers. The parsing rules in a GP parser use Regex patterns, and it is recommended that you have preliminary knowledge of Regex patterns. GP parsing rules have been designed to handle a large set of such complex rules. You cannot create a new data model or edit a built-in data model that uses the GP parser. However, you can duplicate a built-in data model, and then edit its parsing rules as required.

Usage: GP parser-based data models

Sometimes, if the format or pattern of the logs received in LogLogic LMI are slightly different than expected, then the logs are not parsed with the parsing rules defined in the built-in data model. Before LogLogic LMI version 6.4.0, you could not identify which parsing rules failed to match, or which rules were the closest match for such logs, or to edit the parsing rules expressions to accommodate the parsing of the changed log format.

Starting version 6.4.0, you can:

• View a graphical representation of the entire node tree of parsing rules.
• Identify or filter out rules sharing a given node.
• Validate a sample log event against the built-in parsing rules and inspect the graphical view to identify at which node the parsing failed, or the closest matching rule that you might need to tweak.
• Edit the rule expression as required, and save the modified rule or save it as a new rule. See Editing GP Parser-Based Data Models.

List and Graphical Views of GP Parsing Rules

GP parsing rules are displayed as a list on the Rule list tab and as a node tree on the Node tree tab.

Rule list tab

The Rule list tab displays a list of rules as a table. Each row in the table displays the rule number, its rule ID, and its Regex pattern. After clicking the Edit icon , the page displays a table that includes the rule ID and pattern; followed by basic rules and the event descriptor fields. To reorder (move up or down), edit, duplicate, or delete a rule, click the corresponding icon in the rule row.

Node tree tab

The Node tree tab displays the list of rules as a tree. Initially, in the node tree, the first node is green and all others are gray. After parsing a sample log that you provide in the Sample event field, the colors of the nodes change - depending on whether the node was visited; or whether the node pattern matched or failed.

The state of each node is displayed in a tooltip box after hovering over the node and also represented by the following colors:

• Grey: The node has not been visited during the parsing process
• Green: The node has been visited and matched
• Red: The node has been visited but pattern matching failed

Note:

• The colors red and green reflect the node parsing status when a sample log has been provided.
• In the collapsed state, the colors of the nodes are darker than in the expanded state. For example, dark green in collapsed state; but light green in expanded state.

In the node tree area, you can perform the following actions:

• To view node information: If you hover on a node, the node ID, pattern, and rule pattern of that node are displayed as a tooltip box.
• To zoom in or out of the tree: Move the mouse roller or use the mouse trackpad in the tree area.
• To move the tree: Click the background and drag the tree area (keep the mouse button pressed while dragging).
• To filter rules by node: Click the tooltip box of a node. The view switches to the Rule list tab; displaying a filtered list of rules. To clear the filters and view the entire rule list, click the Refresh icon .
• To collapse or expand a node: Click the node. This action triggers an update of the tree layout and changes the view of the node tree. Expanding a node zooms in on the tree; collapsing a node zooms out and might display the entire tree in compact form.

Note: In the collapsed state, the colors of the nodes are darker than in the expanded state. For example, dark green in collapsed state; but light green in expanded state.

Editing GP Parser-Based Data Models

Starting from LogLogic LMI 6.4.0, you can edit the parsing rules of data models that use the GP parser. Because these are built-in data models, you cannot edit them directly. You must duplicate a built-in data model, save it in the User group or any other user-created group, and then edit the parsing rules of your new data model.

Similarly, you cannot create a new GP parsing rule. You can edit an existing rule, and then save it either as the same rule or as a new rule (the new rule is automatically assigned a new rule ID). By default, the GP parsing rules are enabled, and cannot be disabled.

Editing parsing rules involves the following steps:

1. Defining a source filter.

   This procedure is the same as that for advanced data models. See Defining a Source Filter.

2. Editing GP Parser Rules
3. Reviewing the Data Model Configuration

Editing GP Parser Rules

To edit a GP parser rule, you must add a sample event and then perform the following steps.

Procedure
1. In the Sample events box, paste a sample event.

   This data can help in defining the parsing rule based on the log source. After saving the data model, the sample data is always available when editing the same data model or associated parsing rules.

   
   Note:

   • You can paste a maximum of 100 KB of sample data.
   • The tree view shows the node status only for the first event in the Sample events box.
2. Click the Refresh icon . The Parser preview panel displays all extracted columns and their data types that are matched by the corresponding parsing rule.
   
3. Click the Node tree tab to see the tree view. The node tree displays nodes with different colors:
   • Grey: The node has not been visited during the parsing process
   • Green: The node has been visited and matched
   • Red: The node has been visited but pattern matching failed
   
   Note:

   • The colors red and green reflect the node parsing status when a sample log has been provided.
   • In the collapsed state, the colors of the nodes are darker than in the expanded state. For example, dark green in collapsed state; but light green in expanded state.
4. To zoom in or out of the view, move the mouse roller till you have a better view of the matching node. To view the node information of a node, hover over the node.
   
5. After clicking the tooltip box, the view switches to the Rule list tab, and only those parsing rules that share the node you clicked are displayed. Scroll to the rule you want to edit and click its Edit or duplicate icon .
   
6. In the Edit or duplicate rule section, you can edit any fields of the parsing rule except the Rule ID field. For detailed information about the syntax, description, and format rules, see the Field Reference section.
   
7. After modifying the required fields, click Validate. If validation of the rule is successful, a message is displayed at the top of the section. If not successful, an error message is displayed, providing the reason for failure.
   
8. You can either save the changes in the same rule or save the rule as a new rule (duplicate the rule):
   • To save the changes in the same rule, click the Save icon .
   • To save a copy of the rule, click the Save as icon .
   • To cancel the changes, click the Cancel icon .
   Note:

   • The Save and Save as icons are enabled only after making changes in at least one field (except the Rule ID field) and if validation is successful.
   • Saving the parsing rule also saves the data model. 

Reviewing the Data Model Configuration

From the Review configuration page, you can update columns and data types for the associated data model. You can also review column statistics for each defined parsing rule.

Procedure
1. In the Columns panel, all system and custom columns are displayed.

   Field	Description
   Name	The name of the column that is displayed in the results.
   Type	The data type of the column.
   Parser rules	The rule name that includes the defined column.

2. Select the Show system columns check box to show all system columns. By default, some system columns are selected. If the check box is not selected, only the user-defined columns and some default system columns are displayed. For a list of system columns, see Types of Columns.
3. The Match statistics panel displays overall information about events that are matched by specified parsing rules. It displays how many rules are enabled, how many columns are extracted by the rule, and how many events are matched with each rule.

Field Reference

This section explains the fields in a GP parser. You can edit any fields of the parsing rule except the Rule ID field. For information about how to edit these fields, see the Editing GP Parser Rules section.

A GP parsing rule is a sequence of pattern-matching nodes that parse a log. Each rule is made of pattern-matching nodes and event descriptors. Each node includes a set of basic rules. The basic rules contain token definitions, which are the patterns to be used to match and capture substrings from the log. Such substrings are then assigned to the specified database columns. The definition of which substrings are stored in which database columns is stored in the Event Descriptor field. If a log matches the entire node sequence, an event is generated as defined in the Event descriptor field.

The following table describes the fields in a GP parsing rule:

Event Descriptor Language Reference

The event descriptor language is used in the parser rule file. Similar to other programming languages, it includes declarations, statement types, array constructs, and so on.

The following table describes the terms used in this section.

Basic Operations

This section describes the syntax and format of the following operations:

• Declare a field.
• Declare a temporary variable.
• Assign a value to a field or variable.
• Use conditional statements. Iterate over an array.
• Generate multiple events for a log.

TYPE field IDENT;

int field ll_eventID;

temp IDENT <ARRAY_MARKER>; 

IDENT  := expression;

if ( expression ) 
  { statements } 
( else ( if_statement | ({ statements } ) )  )?

Foreach my_var in myTvar { 
   statements 
   }

Foreach my_var in myTvar { 
    //create output data : set fields;  
    insert; 
}

Foreach my_var in myTvar { 
    //create output data : set fields;  
    insert; 
}

Functions

The following table describes a few functions, their syntax, and usage information.

<Data model name>_mapRuleAction(Rule_ID) 

<Data model name>_mapRuleStatus(RuleID_expression) 

foreach element in T_elementList {
   //assign fields values.... ; 
   Insert;
   }

Time format specifiers

Use the following letters to specify the time format parameter in the strptime() function.