System-Defined Baseline Reports

The following system provided baseline reports are continuously running in the system.

Network Traffic Analysis

Performance / Availability Monitoring

Logon Activity

Network Traffic Analysis

Performance / Availability Monitoring

Logon Activity

Reports

You can think of reports as saved or pre-defined versions of searches that you can load and run at any time. AccelOps includes over 2000 pre-defined reports that you can access in Analytics > Reports. Topics in this section describe how to access and view information about reports, how to create baseline reports, and how to use specialized reports like the Identity and Location report. You can refine the results of your reports in the same way that you would refine the results of an historical search or a real time search.

Baseline Reports

System-Defined Baseline Reports

Creating a Report or Baseline Report

Identity and Location Report

Report Bundles

Creating a Report Bundle

Running a Report Bundle

Running System and User-Defined Reports and Baseline Reports

Scheduling Reports

Viewing Available Reports

Baseline Reports

How AccelOps Sets Baselines

Evaluating Rules and Detecting Deviations

When you are setting up AccelOps to monitor your IT infrastructure, you may want to define what is “normal” activity within your systems, and have incidents triggered when a a deviation from that normal activity occurs. For example, you can always assume that there will be some logon failures to a server on a daily basis. Rather than creating a rule that will trigger an incident when a certain hard-coded number of failures occurs, you can set up baseline reports that will trigger an incident when the total number of logon failures over a time period is twice the average over the same time period, or when the deviation from the average is threee times the standard deviation over a specific time period.

By creating a baseline report, you can set mean and standard deviations for any metric and use them in rule, and AccelOps will evaluate the current monitored values against the mean and standard deviation for that time period.

How AccelOps Sets Baselines

Establishing a baseline means recognizing that data center resource usage is time dependent:

Usage is different during weekdays and weekends, and may also be different depending on the day of the week or month Usage is dramatically higher during business hours, typically 8am-5pm

AccelOps maintains distinct baselines for weekdays, weekends and for each hour of day – a total of 24*2 = 48 buckets. Baselines for days of the week or month are not maintained to save memory usage, as this would require 31*24 = 1764 buckets, a 15 fold-increase of memory.

A baseline report is a set of Keys that represent the baselined metrics, and a collection of Values. You can see examples of these Keys and Values in the System-Defined Baseline Reports. These are then used in this process to build the report:

1. During the current hour, the Supervisor and any Worker nodes operate in parallel to save a baseline report in memory by analyzing the report events as a stream.
2. When the hour finishes:
   1. The report is written to disk (on NFS for AccelOps cluster).
   2. The Supervisor module summarizes individual baseline reports from all nodes and forms the baseline for the current hour. c. The baselines are stored in a SQLite database on a local Supervisor.
   3. The Supervisor module reads the previous baseline for the current time interval from the SQLite database. Then it combines the previous values with the current values to create a new baseline.
   4. The new baseline is then stored in SQLite database.
3. For the new hour, a new baseline is created following this process

As this process illustrates, baselining is continuous in AccelOps, and new baseline values are learned adaptively.

Evaluating Rules and Detecting Deviations

A baseline rule contains expressions that involve using the functions STAT_AVG() and STAT_STDDEV() to set dynamic thresholds.

These examples show how STAT_AVG() and STAT_STDDEV() would be used to evaluate the conditions for the example of logon failures in the introduction to this topic.

When AccelOps processes these rules:

1. Rule engine computes the current values in memory.
2. Every 5 minutes:
   1. It looks for STAT_AVG(X:ID) and STAT_STDDEV(X:ID) in memory
   2. If it fails, it retrieves them from the SQLite database and caches them for future use during the hour. c. Evaluates the rule conditions

A sample rule condition involving statistical functions is shown below with (X = AVG(fwConnCount); ID = 112).

Viewing Rules

AccelOps includes a large set of rules for Availability, Performance, Change, and Security incidents in addition to the rules that you can define for your system.

1. To view all system and user-defined rules, go to Analytics > Rules.
2. For multi-tenant deployments, use the Organizations menu in the upper-right corner of the Rules List pane to filter rules by organization.
3. Select any rule in the Rules List to view information about it.

All rules have three information tabs:

Using Watch Lists as Conditions in Rules and Reports

You may want to create a rule that refers to the attributes in a watch list, for example if you want to create a condition in which a Source IP listed in your DNS Violators watch list will trigger an incident.

1. Go to the rule or report where you want to use the watch list.
2. Under Conditions for the report, or under Filters in your rule subpattern, enter the watch list attribute you want to filter for in the Attribut e

For example, Source IP.

3. For Operator, select IN.
4. Click … next to Value, and use the CMDB Browser to find and select the watch list you want to use.

For example, DNS Violators.

5. Click Folder >> to select the watch list, and then click OK.
6. Continue with creating your search criteria or rule sub pattern as you normally would.

Using Geolocation Attributes in Rules

In the same way that you can use geolocation attributes in searches and search results, you can also use them in creating rules. AccelOps includes four system-level rules based on geolocation attributes:

Failed VPN Logon from Outside My Country

Successful VPN Logon from Outside My Country

Large Inbound Transfer From Outside My Country

Large Outbound Transfer To Outside My Country

This screenshot shows the sub pattern for Failed VPN Logon from Outside My Country as an illustration of the way you can use geolocation attributes in a rule.

Setting Global and Per-Device Threshold Properties

Overview

Defining a Global Threshold Property

Defining Per-Device Threshold Properties

Using the DeviceToCMDBAttr Function in a Rule

Overview

In many cases when you create a rule, you set values for device thresholds that should trigger an incident. The example of a rule with a single sub-pattern, for example, contains a condition where if the average CPU utilization of a server exceeds 95% over 3 samples, an incident should be triggered. This is an example of setting an absolute value for the threshold in the rule itself.

Instead of setting an absolute value for the threshold, you can define global threshold properties that you can use as functions within a rule, and also define these threshold properties on a per-device basis. The advantage of this approach is that if you want to change the threshold values in a rule, you can edit the threshold property, rather than having to edit the rule. This is accomplished by using the DeviceToCMDBAttr function to return the value set for that device in the rule.

This table illustrates the difference between using an absolute value, shown in the first column, and threshold property, shown in the second column, in the aggregation conditions for a rule. For the threshold property, the function takes the form of DeviceToCMDBAttr(Host IP,

Threshold Property), while it takes the form of DeviceToCMDBAttr(Host IP, Component, Threshold) for devices with components as shown in the second example.

In the first example, when the rule evaluates the function, the Server CPU Critical rule will return the value of Server CPU Util Critical

Threshold for the host IP if that has been defined for the reporting device, otherwise the global threshold value will return. In the second example, if the Disk Space Util Critical Threshold is defined for a (Host IP,Disk Name) tuple, then the function returns that value, otherwise the global threshold value returns. This is an example of a Map threshold, in which there is one threshold value for each component, and which apply only to disk and interface components.

Defining a Global Threshold Property

AccelOps includes over 30+ pre-defined global threshold properties that you can edit and use in rules, but you can also create custom threshold properties.

1. Go to Admin > Device Support.
2. Click the Custom Properties
3. Click Add.
4. Enter a Name and Display Name for the new threshold property.
5. Enter the Default Value for the threshold.
6. Select the Type of threshold value.

For most global threshold values you will select Double. For Map thresholds, which apply to disks and interfaces, select the Item Type fo r the threshold value, and then select the Component Type to which it applies.

7. Click Save.

Defining Per-Device Threshold Properties

1. Go to CDMB > Devices.
2. Select a device.
3. In the Device Details pane, click Edit.
4. Click the Properties
5. For any of the threshold properties, enter a value.

If you want to edit a Map property, click Edit next to the property name, and then enter the value. If that device does not have any components to which that property could apply, you will see an error message.

6. Click OK.

Using the DeviceToCMDBAttr Function in a Rule

Using the example of the Server CPU Critical rule, you would use the DeviceToCMDB function to set a threshold for the aggregation conditions of the rule in this way:

1. In the sub pattern of the rule, under Aggregation Conditions, click the expression builder icon next to the Attribute
2. In the expression builder, under Add Function, select AVG.
3. In the Add Event Attribute field, select CPU Utilization.
4. Click OK.

The expression builder will close, and you will see the function and event attribute you selected listed as the Attribute for the Aggregate Conditions.

5. For Operator, select =.
6. Click the expression builder icon next to the Value
7. In the Add Function menu, select DeviceToCMDBAttr.
8. In the Select Function Pattern dialog, select DeviceToCMDBAttr(EventAttr,CMDBAttr).
9. Under Add Event Attribute, select Host IP.
10. Under Add CMDB Attribute, select Server CPU Util Critical Threshold.
11. Click OK.
12. Click Save.

Setting Rules for Event Forwarding

In systems management, many servers may need access to forward logs, traps and Netflows from network devices and servers, but it is often resource intensive for network devices and servers to forward logs, traps and netflows to multiple destinations. For example, most Cisco routers can forward Netflow to two locations at most. However, AccelOps can forward/relay specific logs, traps and Netflows to one or more destinations. If you want to send a log to multiple destinations, you can send it to AccelOps, which will use an event forwarding rule to send it to the desired locations.

1. Log in to your Supervisor node.
2. Go to Admin > General Settings > Event Handling.
3. Under Event Forwarding Rule, for multi-tenant deployments, select the organization for which the rule will apply.
4. Click Add.
5. For Sender IP, enter the IP address of the device that will be sending the logs.
6. For Severity, select an operator and enter a severity level that must match for the log to be forwarded.
7. Select the Traffic Type to which the rule should apply.

The Forward To > Port field will be populated based on your selection here.

8. For Forward to > IP, enter the IP address to which the event should be forwarded.
9. Click OK.

Setting Rules for Event Dropping

Some devices and applications generate a significant number of logs, which may be very verbose, contain little valuable information, and consume storage resources. You can configure Event Dropping rules that will drop events just after they have been received by AccelOps, preventing these event logs from being collected and processed. Implementing these rules may require some thought to accurately set the event type, reporting device type, and event regular expression match, for example. However, dropped events do not count towards licensed Events per Second (EPS), and are not stored in the Event database. Dropped event also do not appear in reports, and do not trigger rules. You can also specify that events should be dropped but stored, so event information will be available for searches and reports, but will not trigger rules. And example of an event type that you might want to store but not have trigger any rules would be an IPS event that is a false positive.

Procedure

1. Log in to your Supervisor node.

For multi-tenant deployments you should log in to the Super/Global account if you want to set a system-wide event dropping rule. If you want to set an event-dropping rule for a specific organization, either log in as an administrator for that organization, or or log in using the Super/Global Account and then select the organization to which the rule should apply when you are creating it.

2. Go to Admin > General Settings > Event Handling.
3. Under Event Dropping Rule, click Add.
4. Next to Reporting Device, click Edit, and use the CMDB Browser to find device group or individual device that you want to create the rule for.
5. Next to Event Type, click Edit, and use the Event Type Browser to find the group of event types, or a specific event type, that you want to create the rule for.
6. If the event type you select has an Source IP or Destination IP attribute, you can enter specific IP addresses to which the rule should apply.
7. For Regex Filter, enter any regular expressions you want to use to filter the log files.

If any matches are made against your regular expression, then the event will be dropped.

8. For multi-tenant deployments, select the Organization to which the rule should apply.
9. Select the Action that should be taken when the event dropping rule is triggered.
10. Enter any Description for the rule.
11. Click Save.

Notes

1. All matching rules are implemented by AccelOps, and inter-rule order is not important. If you create a duplicate of an event dropping rule, the first rule is in effect.
2. If you leave a rule definition field blank, then that field is not evaluated. For example, leaving Event Type left blank is the same as selecting All Event Types.
3. AccelOps drops the event at the first entry point. If your deployment uses Collectors, events are dropped by the Collectors. If your deployment doesn’t use Collectors, then the event will be droppedby the Worker or Supervisor where the event is received.
4. You can use the report System Event Processing Statistics to view the statistics for dropped events. When you run the report, select AVG(Policy Dropped Event Rate(/sec) as one of the dimensions for Chart For to see events that have been dropped to this policy.