Adding an Aggregation Rule

When a new aggregation rule is created, it starts computing the metrics for the given GROUP BY expressions. It aggregates data for the events starting from the aggregation start period up to the rule creation time. When an aggregation query matches an aggregation rule, the query results are fetched from the collected aggregation data. When the aggregation rule is created, a filter Blok is created automatically for that rule and is used in future searches.
Attention:
  • When creating aggregations for small periods, for example using GROUP BY minutes or seconds, use the data type as long for the columns in your custom data model.
  • When using your own data model to create aggregation rules, if you edit any column types of the data model after creating an aggregation rule, then for the changes to take effect you must either create a new aggregation rule again, or edit and save the existing aggregation rule.
    Note: Editing an aggregation rule causes all existing data to be purged, and the rule starts computing the aggregation using the updated query.

Procedure

  1. Go to Management > Advanced Features > Rules and click the Aggregation tab.
  2. From the Aggregation page, click to add a new aggregation rule.
    The Add Aggregation Rule dialog box is displayed.
  3. In the Created by field, the name of the user who is currently logged in is displayed automatically.
  4. Enter the rule name in the Name field.
    The name can include letters, numbers, hyphen, or underscore (_).
  5. (Optional) Enter the description of the rule in the Description field.
  6. Enter the search query in the Query field.
    Make sure you enter a valid syntax of a search query. An EQL or SQL query that contains GROUP BY statement and aggregated projections are supported. For details, see GROUP BY Statement. Note the following exceptions:
    • Time functions in the GROUP BY statement can only have sys_eventTime as an argument. Any other timestamp column is not supported.
    • Restrictions on GROUP BY statement:
      • GROUP BY statement can include both column names or expressions. However, only time functions are supported, for example, days(sys_eventTime), weeks(sys_eventTime). Generic expressions such as length(Process)/10 are not supported.
      • Special characters in column names should be escaped, for example, [Process Name] or [% Memory]
    • Restrictions on the projection clause (COLUMNS):
      • The projection items must contain aggregate functions such as sum(), avg().
      • Projection items can be a complex expression, for example, concatenate(max(cpu), '_vs_', min(cpu)), but it should contain aggregation functions. An aggregation function can contain only simple functions as arguments. For example, max(Memory) is supported but max(sqrt(Memory)) is not.

    Click Validate to verify the search statement.

  7. Select or clear the appropriate Aggregation time check boxes to add or remove time functions in GROUP BY clause of the query.
    When you select this option, the time aggregation you selected is inserted in the GROUP BY clause of the rule query. The default time aggregation functions are:
    • weeks(sys_eventTime)
    • days(sys_eventTime)
    • hours(sys_eventTime)
    • minutes(sys_eventTime)
  8. To activate the aggregation rule, click the Enabled slider to ON.
    The optimization starts only after the rule is enabled. The disabled rule does not compute real-time events.
  9. Enter the retention time in the Retention period field for which the computed aggregation values remain stored in the aggregated data.
    You can search based on the same aggregation functions until the specified time has passed. By default, it is set to -1w (1 week).
    For example, if the retention period is -2w (2 weeks), then pre-computed results remain in the system for 2 weeks since the time of rule creation.
  10. In the Compute aggregation for field, specify the time when the aggregation computing must start. A one-time calculation of aggregation happens on the data that was already collected prior to the creation of the aggregation rule. To start computing as soon as the rule is created, leave the field empty. To compute aggregation for the data that exists prior to rule creation, specify the time relative to rule creation time. For example, -1d, -1w, and so on.
    Note:
    • The Compute aggregation for period cannot be more than the retention period.
    • To modify the Compute aggregation for period, first modify the retention period and then the Compute aggregation for period.

    The data model LogLogic_Aggregation_Rule_Metrics provides a disk usage statistics of all aggregation rules in the system. You can run the query USE Loglogic_Aggregation_Rules_Metrics and view the results.

  11. In the Maximum Aggregated Data Storage Size in MB field, enter the maximum storage size of aggregated data. The minimum value is 1 MB, the maximum value is 2147483647, and the default is 1024 MB if the field is empty or 0. You can specify the limit based on the storage availability in your setup. Data is purged daily, based on the retention period and the maximum storage values. Purging is triggered 24 hours after rule creation time. When one of these values is exceeded, the oldest 24 hours data is purged to match the values that you specified in the aggregation rule. For example, with the retention period as -1w and maximum storage size as 1024 MB, suppose that the aggregation data on the first day is 1000 MB and on the second day, is 2024 MB. On the second day, the oldest 24 hours data (in this example, 1000 MB data of the first day) is purged to match the storage size limit of 1024 MB.
  12. Click Save to add the new aggregation rule.
    The newly added rule is displayed on the Aggregation page.

Saving an Advanced Search Query as an Aggregation Rule

If the query you type on the Advanced Search page contains a GROUP BY clause, you can save the query as an aggregation rule.

If the query is not already included in any saved aggregation rule, then the Optimize icon is displayed.

Procedure

  1. Click the optimize icon to open the Add Aggregation Rule dialog box.
    The query that you typed on the Advanced Search page is automatically filled in the Query field.
  2. Enter the other fields in the Add Aggregation Rule dialog box as per Step 4 onwards and then click Save.
    The aggregation rule is saved in the system.