Custom Parsers

To start creating a custom parser for device logs, you should begin by reviewing the Event Parser XML Specification. Writing the XML specification is the primary task in creating a custom parser.

Event Parser XML Specification

Custom Parser XML Specification Template

Parser Name Specification

Device or Application Type Specification

Format Recognizer Specification

Pattern Definition Specification

Parsing Instructions Specification

Creating a Custom Parser

Deleting or Disabling a Parser

Exporting a Custom Parser

Importing a Custom Parser

Parser Examples

Cisco IOS Syslog Parser

 

Event Parser XML Specification

FortiSIEM uses an XML-based parser framework to parse events. These topics describe the parser syntax and include examples of XML parser specifications.

Custom Parser XML Specification Template

Parser Name Specification

Device or Application Type Specification

Format Recognizer Specification

Pattern Definition Specification

Parsing Instructions Specification

Custom Parser XML Specification Template

The basic template for a custom parser XML specification includes five sections. Click on the name of any section for more information.

Section	Description
Parser Name Specification	Name of the parser file
Device Type	The type of device or application associated with the parser
Format Recognizer Specification	Patterns that determine whether an event will be parsed by this parser
Pattern Definition Specification	Defines the parsing patterns that are iterated over by the parsing instructions
Parsing Instructions Specification	Instructions on how to parse events that match the format recognizer patterns

Custom Parser XML Specification Template

Parser Name Specification

This section specifies the name of the parser, which is used only for readability and identifying the device type associated with the parser.

Device or Application Type Specification

This section specifies the device or the application to which this parser applies. The device and application definitions enable FortiSIEM to detect the device and application type for a host from the received events. This is called log-based discovery in FortiSIEM. Once a received event is successfully parsed by this file, a CMDB entry is created with the device and application set from this file. FortiSIEM discovery may further refine the device.

There are two separate subsections for device and application. In each section, vendor, model and version can be specified, but version is not typically needed.

Examples of Specifications for Types of Device and Applications

Hardware Appliances

In this case, the type of event being parsed specifies the device type, for example Cisco IOS, Cisco ASA, etc.

Software Operating Systems that Specify the Device Type

In this case, the type of events being parsed specifies the device type, for example Microsoft Windows etc. In this case the device type section looks like

Applications that Specify Both Device Type and Application

In this case, the  events being parsed specify the device and application types because Microsoft SQL Server can only run on Microsoft Windows OS.

Applications that Specify the Application Type but Not the Device Type

Consider the example of an Oracle database server, which can run on both Windows and Linux operating systems. In this case, the device type is set to Generic but the application is specific. FortiSIEM depends on discovery to identify the device type.

Format Recognizer Specification

In many cases, events associated with a device or application will contain a unique pattern. You can enter a regular expression in the Format Recognizer section of the parser XML file to search for this pattern, which, if found, will then parse the events according to the parser instructions. After the first match, the event source IP to parser file map is cached, and only that parser file is used for all events from that source IP. A notable exception is when events from disparate sources are received via a syslog server, but that case is handled differently.

While not a required part of the parser specification, a format recognizer can speed up event parsing, especially when one parsing pattern file among many pattern files must be chosen. Only one pattern check can determine whether the parsing file must be used or not. The other less efficient option would be to examine patterns in every file. At the same time, the format recognizer must be carefully chosen so that it is not so broad to misclassify events into wrong files, and at the same time, not so narrow that it fails at classifying the right file.

Format Recognizer Syntax

The specification for the format recognizer section is:

In the regexpattern block, a pattern can be directly specified using regex or a previously defined pattern (in the pattern definition section in this file or in the GeneralPatternDefinitions.xml file) can be referenced.

Example Format Recognizers

Cisco IOS

All Cisco IOS events have a %module name pattern.

Cisco ASA

All Cisco ASA events have the pattern ASA-severity-id pattern, for example ASA-5-12345.

Palo Alto Networks Log Parser

In this case, there is no unique keyword, so the entire message structure from the beginning to a specific point in the log must be considered.

Event

<14>May 6 15:51:04 1,2010/05/06 15:51:04,0006C101167,TRAFFIC,start,1,2010/05/06

15:50:58,192.168.28.21,172.16.255.78,::172.16.255.78,172.16.255.78,rule3,,,icmp,vsys1,untrust,untrust,ether net1/1,ethernet1/1,syslog-172.16.20.152,2010/05/06

15:51:04,600,2,0,0,0,0,0×40,icmp,allow,196,196,196,2,2010/05/06 15:50:58,0,any,0

Pattern Definition Specification

In this section of the parser XML specification, you set the regular expression patterns that that FortiSIEM will iterate through to parse the device logs.

You can also write a long pattern definition in multiple lines and indicate their order as shown in this example. The value of the list attribute should be begin in first line and end in last line. If there are more than two lines, the attribute should be set to continue for the other lines.

Parsing Instructions Specification

This section is the heart of the parser, which attempts to recognize patterns in a log message and populate parsed event attributes.

In most cases, parsing involves applying a regular expression to the log, picking up values, and setting them to event attributes. Sometimes the processing is more involved, for example when attributes need to be stored as local variables and compared before populating the event attributes. There are three key components that are used in parsing instructions: Event attributes and variables, inbuilt functions that perform operations on event attributes and variables, and switch and choose branching constructs for logical operations. Values can be collected from both unstructured and structured strings in log messages.

Event Attributes and Variables

Setting an Event Attribute to a Constant

Setting an Event Attribute from Another Variable

Inbuilt Functions

Combining Two or More Strings to Produce a Final String

Normalize MAC Address

Compare Interface Security Level

Convert Hex Number to Decimal Number

Convert TCP/UDP Protocol String to Port Number

Convert Protocol String to Number

Convert Decimal IP to String

Convert Host Name to IP

Add Two Numbers

Divide Two Numbers

Scale Function

Extract Host from Fully Qualified Domain Name

Replace a String Using a Regular Expression

Replace String in String

Resolve DNS Name

Convert to UNIX Time

Trim Attribute

Branching Constructs

Choose Construct

Switch Construct

Collecting Values from Unstructured Strings

Collecting Fields from Structured Strings

Key=Value Structured Data

Value List Structured Data

Event Attributes and Variables

The dictionary of event attributes are defined in FortiSIEM database and any member not belonging to that list is considered a local variable. For readability, local variables should begin with an _, although this is not enforced.

Setting an Event Attribute to a Constant

Setting an Event Attribute from Another Variable

The $ symbol is used to specify the content of a variable. In the example below, attribute hostMACAddr gets the value stored in the local variable

Combining Two or More Strings to Produce a Final String

This is accomplished by using the combineMsgId function. Here _evIdPrefix is the prefix, _evIdSuffix is the suffix, and the output will be s tring1-_evIdPrefix-_evIdSuffix.

Normalize MAC Address

This is accomplished by using the normalizeMAC function. The output will be six groups of two nibbles separated by a colon, for example AA:BB

This is accomplished by using the compIntfSecVal function. This primarily applies to Cisco ASA and PIX firewalls. The results returned are:

This is accomplished by using the convertHexStrToInt function.

Convert TCP/UDP Protocol String to Port Number

This is accomplished by using the convertStrToIntIpPort function.

Convert Protocol String to Number

This is accomplished by the using the convertStrToIntIpProto function.

Convert Decimal IP to String

This is accomplished by using the converIpDecimalToStr function.

Convert Host Name to IP

This is accomplished by using the convertHostNameToIp function.

Add Two Numbers

This is accomplished by using the add function.

Divide Two Numbers

This is accomplished by using the divide function.

Scale Function

This is accomplished by using the scale function.

Extract Host from Fully Qualified Domain Name

This is accomplished by using the extractHostFromFQDN function. If _fqdn` contains a . , get the string before the first .,  otherwise, get the whole string.

Replace a String Using a Regular Expression

This is accomplished by using the replaceStringByRegex function.

Replace String in String

This is accomplished by using the replaceStrInStr function.

Resolve DNS Name

This is accomplished by using the resolveDNSName function, which converts DNS name to IP address.

Convert to UNIX Time

This is accomplished by using the toDateTime function.

Trim Attribute

This is accomplished by using the trimAttribute function. In the example below, it is used to trim the leading and trailing dots in destName.

Branching Constructs

Choose Construct

The format is:

Switch Construct The format is:

Collecting Values from Unstructured Strings

From a string input source, a regex match is applied and variables are set. The variables can be event attributes or local variables. The input will be a local variable or the default raw message variable. The syntax is:

The regexpattern is specified by a list of variables and sub-patterns embedded within a larger pattern. Each variable and sub-pattern pair are enclosed within <>.

Consider an example in which the local variable _body is set to list 130 permitted eigrp 172.16.34.4(Serial1 ) > 172.16.34.3, 1 packet. From this sting we need to set the values to local variables and event attributes.

Value	Set To	Type
130	_aclName	Local Variable
permitted	_action	Local Variable
eigrp	_proto	Local Variable
172.16.34.4	srcIpAddr	Event Attribute
Serial1	srcIntfName	Event Attribute
172.16.34.3	destIpAddr	Event Attribute
1	totPkts	Event Attribute

This is achieved by using this XML. Note that you can use both the collectAndSetAttrByRegex and collectFieldsByRegex functions to collect values from fields.

Collecting Fields from Structured Strings

The are usually two types of structured strings in device logs:

Key=value structured

Value list structured

In each case, two simpler specialized parsing constructs than are provided

Key=Value Structured Data

Certain logs, such as SNMP traps, are structured as Key1 = value1 <separator> Key2 = value2,…. These can be parsed using the col lectAndSetAttrByKeyValuePair XML attribute tag with this syntax.

When a key1 match is found, then the entire string following key1 up to the separatorString is parsed out and stored in the attribute variab leOrEventAttribute1.

As an example, consider this log fragment.

_body =

SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.60 = Hex-STRING: 07 D8 06 0B

13 15 00 00 2D 07 00    SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.11.0

= Hex-STRING: 00 16 B6 DB 12 22

SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.12.0 = Hex-STRING: 00 21 55

4D 66 B0  SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.13.0 = INTEGER: 36

SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.1.0 = Hex-STRING: 00 1A 1E C0

60 7A  SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.56.0 = INTEGER: 2   SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.17.0 = STRING:

“00:1a:1e:c0:60:7a”

The corresponding parser fragment is:

After parsing, the attribute values are set:

Value	Attribute
00 16 B6 DB 12 22	srcMACAddr
00 21 55 4D 66 B0	destMacAddr
2	wlanRadioId
00:1a:1e:c0:60:7a	apMac

Value List Structured Data

Certain application logs, such as those from Microsoft IIS, are structured as a list of values with a separator. These can be parsed using the coll ectAndSetAttrByPos XML attribute tag following this syntax.

When the position offset1 is encountered, the subsequent values up to the separatorString is stored in variableOrEventAttribute1.

As an example, consider this log fragment.

The parser fragment is:

<collectAndSetAttrByPos src=”$_body” sep=’  ‘>

<attrPosMap attr=”srvInstName” pos=’1’/>

<attrPosMap attr=”destName” pos=’2’/>

<attrPosMap attr=”relayDevIpAddr” pos=’2’>

<attrPosMap attr=”destIpAddr” pos=’3’/>

<attrPosMap attr=”httpMethod” pos=’4’/>

<attrPosMap attr=”uriStem” pos=’5’/>

<attrPosMap attr=”uriQuery” pos=’6’/>

<attrPosMap attr=”destIpPort” pos=’7’/>

<attrPosMap attr=”user” pos=’8’/>

<attrPosMap attr=”srcIpAddr” pos=’9’/>

<attrPosMap attr=”httpVersion” pos=’10’/>

<attrPosMap attr=”httpUserAgent” pos=’11’/>

<attrPosMap attr=”httpReferrer” pos=’13’/>

<attrPosMap attr=”httpStatusCode” pos=’15’/>

<attrPosMap attr=”httpSubStatusCode” pos=’16’/>

<attrPosMap attr=”httpWin32Status” pos=’17’/>

<attrPosMap attr=”recvBytes” pos=’18’/>

<attrPosMap attr=”sentBytes” pos=’19’/>

<attrPosMap attr=”durationMSec” pos=’20’/>

</collectAndSetAttrByPos>

For structured strings, techniques in this section are more efficient than in the previous section since, the expression is simpler and ONE tag can be used to parse regardless of the order in which the keys or values appear in the string.

 

 

Creating a Custom Parser

Prerequisites

You should have examples of the logs that you want to parse

You should have created any new device/application types, event attribute types, or event types that you want to use in your XML specification

You should already have written the XML specification for your parser

You should have prepared a test event that you can use to validate the parser

Parsers Applied in Order

Parsers are applied in the order they are listed in Admin > Device Support > Parsers, so it is important to add your custom parser to the list in relation to any other parsers that may be applied to your device logs. If you click Fix Order, this will arrange the parsers with system-defined parsers at the top of the list in their original order, and user-defined parsers at the bottom. By sure to click Apply to make sure the change in order is picked up by the back-end module.

Procedure

1. Go to Admin > Device Support > Parsers.
2. Select a parser that is above the location in the list where you want to add your parser, and then click New.
3. Enter a Name for the parser.
4. Select a Device Type to which the parser should apply.

If the device type doesn’t appear in the menu, you should create a new device type

5. Enter a Test Event containing an example of an event that you want to use to validate the parser.
6. Enter the Parser XML.
7. Click Validate.

This will validate the XML.

8. Click Test.

This will send the test event to the parser to make sure it is parsed correctly, and will also test the parsers above and below yours in the list to make sure they continue to parse logs correctly.

9. If the XML for your parser validates and the test event is correctly parsed, select Enable.

If you need to continue working on your parser, you can Save it without selecting Enable.

10. Click Save.
11. Click Apply to have the backend module pick up your parser and begin applying it to device logs.

You should now validate that events are being parsed by creating some activity that will cause a log to be generated, and then run a query against the new device IP address and validate the parsed results.

 

 

 

Deleting or Disabling a Parser

1. Go to Admin > Device Support > Parsers.
2. Select the parser you want to delete or disable.
3. Click Delete or Disable.
4. Click Yes to confirm that you want to delete or disable the parser.

Exporting a Custom Parser

To export a parser, you must also export XML files for the device/app types, event attribute types, event types, and then the parser specification file used by your parser.

1. Go to Admin > Device Support > Device/App Types.
2. Select the device/application types used in your parser, and then click Export.
3. Go to Admin > Device Support > Event Attribute Types.
4. Select the event attribute types used in your parser, and then click Export.
5. Go to Admin > Device Support > Event Types.
6. Select the event types used in your parser, and then click Export.
7. Go to Admin > Device Support > Parsers.
8. Select the parser specification for your parser, and then click Export.

Importing a Custom Parser

Importing a custom parser involves importing four XML files: the XML files containing any device/app types, event attribute types, or event types that you have created for this parser, followed by the parser specification XML file.

1. For each device/app type, event attribute type, or event type XML file that is required for your parser, go to the appropriate tab in Admin > Device Support, and then click Import.
2. Browse to the location of your XML file, and then click Upload.
3. Go to Admin > Device Support > Parsers, and then click Import.
4. Browse to the location of your parser specification XML file, and then click Upload.
5. Follow the instruction in Creating a Custom Parser to validate your XML and test the parser, and to make sure it appears in the correct position in the list of parsers.

Parser Examples

Cisco IOS Syslog Parser

Cisco IOS Syslog Parser

Add Device Type

Create a file CiscoIOSParser.xml with this content.

Create the Parser Specification and Add Local Patterns

Create the parser XML file with this content, and add the pattern definition patCiscoIOSMod for detecting IOS modules such as SEC.

 

Define the Format Recognizer

Add this format recognizer for detecting %SEC-6-IPACCESSLOGP, which is a signature of Cisco IOS syslog messages.

Parse the Syslog Header

A syslog message consists of a syslog header, and a body. For better organization, we first parse the syslog header and event type. Subsequent code will include event type specific parsing, which is why event type is extracted in this step. In this example, the header is in boldface.

<190>91809: Jan 9 02:38:47.872: %SEC-6-IPACCESSLOGP: list testlog permitted tcp 192.168.20.33(3438) -> 69.147.86.184(80), 1 packet

The XML code for parsing the header does the following:

1. Matches the pattern <190>91809: Jan 9 02:38:47.872: %SEC-6-IPACCESSLOGP:
2. Sets the eventType attribute to IOS-SEC- IPACCESSLOGP.
3. Sets deviceTime.
4. Sets event severity (1-7 scale in Cisco IOS, 1=> most severe, to normalized 1-10 scale in FortiSIEM where 10=>most severe)
5. Saves the event list testlog permitted tcp 192.168.20.33(3438) -> 69.147.86.184(80), 1 packet in a temporary variable _body.

Note that the patterns gPatSyslogPRI, gPatMon, gPatDay, gPatTime, gPatInt, gPatmesgBody are global patterns that are defined in the GeneralPatternDefinitions.xml file:

This parser file XML fragment for parsing the example syslog message looks like this:

 

Parse the Syslog Body

The parsing is done on an eventType by eventType basis, since the formats are eventType specific. Parsing the syslog body involves three steps:

1. Parsing the action string. Based on the action staring value (permit or denied), modify the eventType by appending the action string value at the end, and also modify the eventSeverity
2. Parsing the protocol, source and destination IP, port, and totalPackets.
3. Converting the protocol string to a protocol integer.

 

Final Parser

</patternDefinitions>

<parsingInstructions>

<!—parse header –>

<collectFieldsByRegex src=”$_rawmsg”>

 

<regex><![CDATA[<:gPatSyslogPRI>?<:gPatMon>\s+<:gPatDay>\s+<:gPatTime>

%<_evIdPrefix:patCiscoIOSMod>-<_severity:gPatInt>-<_evIdSuffix:patStrEnd

Colon>: <_body:gPatMesgBody>]]></regex>

</collectFieldsByRegex>

<setEventAttribute attr=”eventType”>combineMsgId(“IOS-“,

$_evIdPrefix, “-“, $_evIdSuffix)</setEventAttribute>

<choose>

<when test=’$_severity IN “6, 7″‘>             <setEventAttribute attr=”eventSeverity”>1</setEventAttribute>         </when>

<when test=’$_severity = “1”‘>            <setEventAttribute attr=”eventSeverity”>10</setEventAttribute>         </when>

<when test=’$_severity = “2”‘>

<setEventAttribute attr=”eventSeverity”>8</setEventAttribute>

</when>

<when test=’$_severity IN “3, 4″‘>

<setEventAttribute attr=”eventSeverity”>5</setEventAttribute>

</when>

<when test=’$_severity = “5”‘>

<setEventAttribute attr=”eventSeverity”>2</setEventAttribute>

</when>

</choose>

<!—parse body –>

<choose>

<when test=’$eventType IN “IOS-SEC-IPACCESSLOGP,

IOS-SEC-IPACCESSLOGDP, IOS-SEC-IPACCESSLOGRP”‘>

<collectAndSetAttrByRegex src=”$_body”>

<regex><![CDATA[list

<_aclName:gPatStr>\s+<_action:gPatWord>\s+<_proto:gPatWord>\s+<srcIpAddr

:gPatIpV4Dot>\(<srcIpPort:gPatInt>\)<:gPatMesgBody>->\s+<destIpAddr:gPat

IpV4Dot>\(<destIpPort:gPatInt>\),\s+<totPkts:gPatInt> <:gPatMesgBody>]]>

</regex>           </collectAndSetAttrByRegex>

<choose>

<when test=’$_action = “permitted”‘>                  <setEventAttribute

attr=”eventType”>combineMsgId(“IOS-“, $_evIdPrefix, “-“, $_evIdSuffix, “-PERMITTED”)</setEventAttribute>           <setEventAttribute attr=”eventSeverity”>1</setEventAttribute>

</when>

<when test=’$_action = “denied”‘>

<setEventAttribute attr=”eventType”>combineMsgId(“IOS-“, $_evIdPrefix, “-“, $_evIdSuffix, “-DENIED”)</setEventAttribute>                  <setEventAttribute attr=”eventSeverity”>3</setEventAttribute>               </when>

</choose>           <setEventAttribute attr=”ipProto”>convertStrToIntIpProto($_proto)</setEventAttribute>

Parsed Output Input syslog:

<190>91809: Jan 9 02:38:47.872: %SEC-6-IPACCESSLOGP: list testlog permitted tcp 192.168.20.33(3438) ->

69.147.86.184(80), 1 packet

Parsed fields:

1. phRecvTime: the time at which the event was received by FortiSIEM
2. phDeviceTime: Jan 9 02:38:47 2010
3. eventType: SEC-IPACCESSLOGP-PERMITTED
4. eventSeverity: 3
5. eventSeverityCategory: LOW
6. aclName: testlog
7. ipProto: 6
8. srcIpAddr: 192.168.20.33
9. destIpAddr: 69.147.86.184
10. srcIpPort: 3438
11. destIpPort: 80
12. totPkts: 1

The master list of event attributes supported by FortiSIEM is here

Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!

Creating Event Attributes, Event Types, and Device Types

When you create a custom parser or monitor, you must also specify the device, application, event type, and event attribute to which it applies. If these objects aren’t already included in the FortiSIEM CMDB, you can create them as a preliminary step to creating your parser or monitor.

Creating Device and Application Types

Creating Event Attribute Types Creating Event Types

Creating Device and Application Types

If the device or application that you want to create a parser or monitor for isn’t already listed in Admin > Device Support > Device/App Types, you can add it.

1. Go to Admin > Device Support > Device/App Types.
2. Click New, and then choose New Device Type or New Application Type.
3. Enter the information for the new device or application type.

Device Type	Vendor Model Version Device/App Group Biz Service group Description
Application Type	Vendor Model Version Device/App Group Biz Service group Application Package Group Description

 

4. Click Save.

Creating Event Attribute Types

Event attributes are used to capture parsed information from events. You only have to create a new attribute if the one you want use for your custom parser or monitor is not listed in Admin > Device Support > Event Attribute Types.

2. Click New.
3. Enter a Name and Display Name.
4. Select the Value Type to associate with the event attribute type.
5. Optionally enter a Display Format Type and Description.
6. Click Save.

 

Creating Event Types

After parsing an event or log, FortiSIEM assigns a unique event type to that event/log. When you create a new custom parser for device logs, you almost always have to add a new event type to FortiSIEM so the log events can be identified.

1. Go to Admin > Device Support > Event Types.
2. Click New.
3. Enter a Name for the new event type.
4. Select the Device Type to associate with the event type.

If the device type isn’t included in the menu options, you can add it to FortiSIEM.

5. Select the Event Type Group category for this event type.
6. Select a Severity to associate with the event type.
7. Enter an optional Description.
8. Click Save.

Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!

Creating Custom Parsers and Monitors for Devices

Creating a custom parser for device logs involves writing an XML specification for the parser, and then using a test event to make sure the logs are parsed correctly. Creating a custom monitor involves defining a performance object that you want to monitor, associating that performance object to a device type, event type, and event attribute type, and then testing to make sure that the monitored metrics are correctly received by FortiSIEM. You can create custom monitors for system and application performance, command outputs, and file monitoring.

Creating a Custom Multi-Line SSH Command Output Monitor

Creating a Custom WINEXE Command Output Monitor

Custom File Monitor

Agent-less File-Integrity Monitoring

Agent-less Target File Monitoring Custom Configuration Change Monitoring

Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!

Data Update Subscription Service

FortiSIEM is constantly developing support for additional IT infrastructure devices. By subscribing to the FortiSIEM Data Update Service, you can receive updates when support for new devices becomes available, rather than waiting for it to be included in a formal release. In addition to devices you can also receive new rules, reports, parser updates etc.

Data Update Overview

Configuring Data Update

Data Update Overview

FortiSIEM data update subscription service updates your FortiSIEM deployment with the latest device support related data as it becomes available, rather than having to wait for it to be included in a formal release.

The following items can be included in an update

New event attribute

New event types

New device type

New parsers or modifications for existing parsers

Performance monitoring templates for new devices or modified ones for existing devices

New rules or modifications for existing rules

New reports or modifications for existing reports – both CMDB report and event based reports

New groups or modifications for existing groups for Event Types, Rules, Reports, Device Groups, Application Groups Code to handle new devices

 

 

Configuring Data Update

Provide a brief (two to three sentence) description of the task or the context for the task.

Prerequisites

Procedure

Configure Data Update Server Setting

Check Available Data Updates

Apply Data Update on Supervisor

Apply Data Update on Collectors

Check whether Data Update Installed Successfully

Prerequisites

Contact FortiSIEM support and make sure that your license includes Data Update Service

Make sure you have Data Update URL – this is typically https://images.FortiSIEM.net/upgrade/ds – contact FortiSIEM to make sure that this information has not changed

Make sure you have license credentials

Procedure

Configure Data Update Server Setting

1. Log on to FortiSIEM Supervisor with Administrator credentials
2. Go to Admin > General Settings > System
3. Configure Data Update Server Setting
   1. Enter Data Update URL (see prerequisites)
   2. Enter Server Username and Server Password – these are the license credentials
   3. Specify Notify Email (optional) – you will receive email when new data updates are available d. Click Save

Check Available Data Updates

1. Log on to FortiSIEM Supervisor with Administrator credentials
2. Go to Admin > Data Update
3. Click Refresh
   1. Available data updates are shown on left
   2. Click a version on the left and the contents for that version is shown on the right
4. Check the current data version from Admin > Cloud Health > Data Update Version. The number after 3rd decimal is the data version. For example 4.4.1.38 means data version is 38.
5. Note the data version you would like to upgrade to.

Apply Data Update on Supervisor

1. SSH to FortiSIEM Supervisor as root
2. Go to /pbin
3. Download the data version by running ./phdownloaddata and specify the data version you would like to upgrade to
4. Install the data version by running ./phinstalldata

Apply Data Update on Collectors

1. Log on to FortiSIEM Supervisor with Administrator credentials
2. Go to Admin > Collector Health
   1. Select a Collector
   2. Click Download Data Update – this downloads the data files to the collector
   3. Click Install Data Update – this installs the data files on the collector
   4. Repeat for all collectors

Check whether Data Update Installed Successfully

1. Log on to FortiSIEM Supervisor with Administrator credentials
2. Check Admin > Cloud Health > Data Update Version
3. Check Admin > Collector Health > Data Update Version

 

 

 

Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!

Creating Business/IT Services

By defining an IT or Business Service, you can create a logical grouping of devices and IT components which can be monitored together.

1. Log in to your Supervisor node.
2. Go to CMDB > Business Services.
3. Click New.
4. Enter a Name and Description for the business service.
5. Select a Device/Application Group, and when the list of associated devices loads into the selection pane, select a device and click >> t o add it to the Selected Devices/Applications for the business service.
6. Click Save when you’re done adding devices to the business service.

After you have created a business service, you can select it, and the Show Topology option, to view it within overall IT topology. You can also use the links in the Analysis menu of the Business Services summary dashboard to find out more information about incidents, device availability, device and application performance, interface and event status, and real-time and historical search for a selected business service.

 

Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!

Configuring Monitoring

Once FortiSIEM discovers your devices, they will monitored continuously, and you can use the data collected to analyze the performance of your infrastructure. You can also configure FortiSIEM to send notifications when events that meet specific conditions occur in your infrastructure.

You can disable the collection of metrics for specific devices, disable devices for monitoring, and change the polling interval for metric collection.

Some devices need to be configured to send logs to FortiSIEM, as described in the topics under Configuring External Systems for Discovery, Monitoring and Log Collection. You can also configure FortiSIEM to monitor important ports, processes, and interfaces, and set up monitoring tests that use synthetic transaction to make sure that critical services are up and running.

Device Monitoring Settings

Adding Important Interfaces

Adding Important Processes

Adding Important Ports

Excluding Disks from Disk Capacity Utilization Monitoring

Managing Monitoring of System and Application Metrics for Devices

Setting Up Synthetic Transaction Monitoring Tests

Protocol Settings for Synthetic Transaction Monitoring Tests Adding a Synthetic Monitoring Test to a Business Service

Device Monitoring Settings

While FortiSIEM constantly monitors and reports on your IT infrastructure, there are several settings you can use to refine reporting on critical interfaces, important processes and ports, and disk utilization.

Adding Important Interfaces

Adding Important Processes

Adding Important Ports

Excluding Disks from Disk Capacity Utilization Monitoring

Adding Important Interfaces

This setting allows you to always get interface utilization reports on a set of important network interfaces across all device types.

Important Interface Setup after 4.8.1 Upgrade

The behavior of interface monitoring has dramatically changed since 4.8. So it is very important to follow these steps.

1. Create a list of all Important interfaces
2. Go to Admin > General Settings > Monitoring > Important Interfaces Click Enable. This will stop all interface monitoring.
3. Click
4. Select either Device View or Interface View.
5. Select a device to view and select its interfaces, or select an interface.
6. Click OK to add the selected interface to the list. The Critical and Monitor boxes would be automatically checked.
7. Check the WAN box if applicable. If checked, the interface utilization events would have isWAN = “yes” attribute. You can use this to run a report for all WAN interfaces.
8. Click Apply All. Now FortiSIEM will start monitoring only the selected interfaces in this tab will be monitored.
9. If you want to disable this behavior and return to ALL interface monitoring (as in releases prior to 4.8), then click Disable.

Adding Important Processes

This setting allows you to always get process resource utilization reports and up/down alerts on a set of important processes across all device types.

Important Process Setup after 4.8.1 Upgrade

The behavior of process utilization monitoring has dramatically changed since 4.8. So it is very important to follow these steps.

1. Create a list of all Important interfaces
2. Go to Admin > General Settings > Monitoring > Important Processes Click Enable. This will stop all interface monitoring.
3. Click
4. Enter a Process Name and any Parameters, and then click OK.
5. Click Apply All. Now FortiSIEM will start monitoring only the selected processes in this tab.
6. If you want to disable this behavior and return to ALl interface monitoring, then click Disable.

Adding Important Ports

Always reporting the UP/DOWN status for every TCP/UDP port on every server can consume a significant amount of resources. FortiSIEM will report the UP/DOWN status only for the ports you add to the Important Ports list. Matching is exact based on port number and IP protocol.

1. Go to Admin > General Settings > Monitoring.
2. Under Important Ports, click Add.
3. Enter the Port Number and select the Port Type.
4. Click OK.
5. Click Apply All.

Excluding Disks from Disk Capacity Utilization Monitoring

You can exclude disks from disk capacity utilization monitoring. Disk capacity utilization events will not be generated for devices matching the device name, access IP, and disk name that you provide. Incidents will not trigger for these events, and the disks will not show up in summary dashboards.

2. Under Excluded Disks, click Add.
3. Select a device to to view its disks, and then select the disk you want to exclude from monitoring.
4. Click OK.
5. Click Apply All.

Managing Monitoring of System and Application Metrics for Devices

When FortiSIEM discovers devices, it also discovers the system and application metics that can be monitored for each device, and displays these in the Monitor Change/Performance tab of the Setup Wizard. Here you can also disable the monitoring of specific metrics for devices, disable devices from being monitored, and change the polling interval for specific metrics. See Inspecting Event Pulling Methods for Devices for an explanation of the different status indicators for System Monitor and Application Monitor metrics.

1. Go to Admin > Setup Wizard > Monitor Change/Performance.
2. Click Refresh to make sure you have the latest list of devices.
3. To disable monitoring for a device, clear the Enable option for it.
4. To enable or disable monitoring of a specific metrics for a device, click on a device to select it, then click Edit and select System Monitoring or Application Monitoring to view the list of metrics associated with that monitor and device.
5. To change the polling interval for a metric, in the More menu, select Set Intervals. Select the Monitor Type and Device, and then set the interval.
6. When you are done making changes, click Apply.

Setting Up Synthetic Transaction Monitoring Tests

A Synthetic Transaction Monitoring (STM) test lets you test whether a service is up or down, and measure the response time. An STM test can range from something as simple as pinging a service, to something as complex as sending and receiving an email or a nested Web transaction. Setting up an STM test involves defining the type of monitor, associating the monitor definition to a device and testing it, and then deploying the STM test to a Supervisor or Collector. You can view the results of STM tests in the Synthetic Transaction Monitoring page, either by navigating to Summary Dashboard > Availability/Performance > Application Summary > Synthetic Transaction Monitoring, or to Admin > Setup Wizard > Synthetic Transaction Monitoring, and then clicking on Monitoring Status. You can also report on the results of STM tests in the reports Top Applications By Synthetic Transaction Response Time and Top Applications By Synthetic Transaction Response Time –

Detailed view. When an STM test fails, three system rules are triggered, and you can receive an email notification of that failure by creating a notification policy for these rules.

System Rule	Description
Service Degraded – Slow Response to STM	Detects that the response time of an end-user monitored service is greater than a defined threshold (average over 3 samples in 15 minutes is more than 5 seconds)
Service Down – No Response to STM	Detects a service suddenly went down from the up state and is no longer responding to synthetic transaction monitoring probes.
Service Staying Down – No Response to STM	Detects a service staying down, meaning that it went from up to down and did not come up, and is no longer responding to end user monitoring probes

1. Go to Admin > Setup Wizard > Synthetic Transaction Monitoring.
2. Click Add.
3. Enter a Name and Description for the test.
4. For Frequency, enter how often, in minutes, you want the test to run.
5. Select the Protocol for your test.

See Protocol Settings for Synthetic Transaction Monitoring Tests for more information about the settings and test results for specific protocols.

6. Click Save.

You now have to associate the STM test with a target host name, IP address, or IP range.

7. Click Create and Test.
8. For Monitoring Definition select one of the STM tests you have created.
9. For Host Name or IP/Range, enter the information for your STM test target.
10. For Port, click + and enter any ports to use when connecting to the target with this test.
11. Click OK.

FortiSIEM will run the test and verify if it is successful. If it succeeds, it will be added to the list of tests with a yellow Star next to it, indicating that it has been added but is not yet running.

12. Click Apply All to begin executing your tests at their set frequency.

The yellow Star will be removed from your test after it executes against the target the first time

 

Protocol Settings for Synthetic Transaction Monitoring Tests

This table describes the settings associated with the various protocols used for setting up Synthetic Transaction Monitoring tests.

Protocol	Description	Settings	Notes
Ping	Checks packet loss and round trip time	Maximum Packet Loss PCT: tolerable packet loss Maximum Average Round Trip Time: tolerable round trip time (seconds) from FortiSIEM to the destination and back If either of these two thresholds are exceeded, then the test is considered as failed.
LOOP Email	This test sends an email to an outbound SMTP server and then attempts to receive the same email from a mailbox via IMAP or POP. It also records the end-to-end time.	Timeout: the time limit by which the end to end LOOP EMAIL test must complete. Outgoing Settings: these specify the outgoing SMTP server account for sending the email. SMTP Server: name of the SMTP server User Name: user account on the SMTP server Email Subject: content of the subject line in the test email Incoming Settings: These specify the inbound IMAP or POP server account for fetching the email. Protocol Type: choose IMAP or POP Server: name of the IMAP or POP server User Name: user account on the IMAP or POP server Email Subject: content of the subject line in the test email	Before you set up the test you will need to have set up access credentials  for an outbound SMTP account for sending email, and an inbound POP/IMAP account for receiving email
HTTP(S) – Selenium Script	This test uses a Selenium script to play back a series of website actions in FortiSIEM.	Upload: select the java file you exported from Selenium Total Timeout: the script must complete by this time or the test will be considered failed Step Timeout: each step must complete by this time	How to export: Make sure Selenium IDE is installed within Firefox browser Open Firefox Launch Tools > Selenium IDE. From now on, Selenium is recording user actions Visit websites Once done, stop recording Click File > Export Test case as > Java / Junit 4 /WebDriver Save the file as .java in your desktop. This file has to be inputted in FortiSIEM.
HTTP(S) – Simple	This test connects to a URI over HTTP(s) and checks the response time and expected results	URI: the URI to connect to Authentication: any authentication method to use when connecting to this URI Timeout: this is the primary success criterion – if there is no response within the time specified here, then the test fails Contains: an expected string in the test results Does Not Contain: a string that should not be contained in the test results Response Code: an expected HTTP(S) response code in the test results. The default is set to 200 – 204.

 

HTTP(S) – Advanced	This test uses HTTP requests to connect to a URI over HTTP(s), and checks the response time and expected results	Click + to add an HTTP request to run against a URI. URI: the URI to run the test against SSL: Whether or not to use SSL when connecting to the URI, and the port to connect on Authentication: the type of authentication use when connecting to the URI Timeout: this is the primary success criterion – if there is no response within the time specified here, then the test fails Method Type: the type of HTTP request to use Send Parameters: click + or the Pencil ic on to add or edit any parameters for the request Contains: an expected string in the test results Does Not Contain: a string that should not be contained in the test results Response Code: an expected HTTP(S) response code in the test results. The default is set to 200 – 204. Store Variables as Response Data for Later Use: click + or the Pencil icon to add or edit any variable patterns that should be used as data for later tests
TCP	This test attempts to connect to the specified port using TCP	Timeout: this is the single success criterion. If there is no response within the time specified here, then the test fails.
DNS	Checks response time and expected IP address	Query: the domain name that needs to be resolved Record Type: the type of record to test against Result: specify the expected IP address that should be associated with the DNS entry Timeout: this is the primary success criterion – if there is no response within the time specified here, then the test fails
SSH	This test issues a command to the remote server over SSH, and checks the response time and expected results	Remote Command: the command to run after logging on to the system Timeout: this is the primary success criterion – if there is no response within the time specified here, then the test fails Contains: an expected string in the test results	You will need to have set up an SSH credential on the target server before setting up this test As an example test, you could set Raw Command t o ls, and then set Contains to the name of a file that should be returned when that command executes on the target server and directory

 

 

LDAP	This test connects to the LDAP server, and checks the response time and expected results	Base DN: an LDAP base DN you want to run the test against Filter: any filter criteria for the Base DN Scope: any scope for the test Timeout: this is the primary success criterion – if there is no response within the time specified here, then the test fails Number of Rows: the expected number of rows in the test results Contains: an expected string in the test results Does Not Contain: a string that should not be contained in the test results	You will need to have set up an access credential for the LDAP server before you can set up this test
IMAP	This tests checks connectivity to the IMAP service	Timeout: this is the single success criterion – if there is no response within the time specified here, then the test fails
POP	This test checks connectivity to the IMAP service	Timeout: this is the single success criterion – if there is no response within the time specified here, then the test fails
SMTP	This test checks connectivity to the SMTP service	Timeout: this is the single success criterion – if there is no response within the time specified here, then the test fails
JDBC	This test issues a SQL command over JDBC to a target database, and checks the response time and expected results	JDBC Type: the type of database to connect to Database Name: the name of the target database SQL: the SQL command to run against the target database Timeout: this is the primary success criterion – if there is no response within the time specified here, then the test fails Number of Rows: the expected number of rows in the test results Contains: an expected string in the test results Does Not Contain: a string that should not be contained in the test results
FTP	This test issues a FTP command to the server and checks expected results	Anonymous Login: choose whether to use anonymous login to connect to the FTP directory Remote Directory: the remote directory to connect to Timeout: this is the primary success criterion – if there is no response within the time specified here, then the test fails
TRACE ROUTE	This test issues a trace route command to the destination and parses the results to create PH_DEV_MON_TRACEROUTE events, one for each hop.	Timeout: If there is no response from the system within the time specified here, then the test fails. Protocol Type: Specifies the IP protocol over which trace route packets are send current options are UDP, TCP and ICMP Max TTL: Max time to live (hop) value used in outgoing trace route probe packets. Wait Time: Max time in seconds to wait for a trace route probe response	For the trace route from AO to destination D via hops H1, H2, H3, FortiSIEM generates 3 hop by hop PH_DEV_MON_TRACEROUTE events. First event: Source AO, destination H1, Min/Max/Avg RTT, Packet Loss for this hop Second event: Source H1, destination H2, Min/Max/Avg RTT, Packet Loss for this hop Third event: Source H2, destination H3, Min/Max/Avg RTT, Packet Loss for this hop Fourth event: Source H3, destination D, Min/Max/Avg RTT, Packet Loss for this hop

Adding a Synthetic Monitoring Test to a Business Service

You may want to add a Synthetic Transaction Monitoring (STM) test to a Business Service as part of the monitoring infrastructure for that service. However, in order to enable reporting on that STM, you need to add it to the business service as a device that FortiSIEM can then report on. This topic explains how to create a device for an STM test and add it to your business service report.

1. Create your STM as described in Setting Up Synthetic Transaction Monitoring Tests.
2. Note the IP address that your STM resolves to in Step 9 of the setup instructions.
3. In the CMDB tab, select Devices, and then select a subcategory where you want to add the STM device.

You may want to create your own group where you manage your STM devices.

4. In the summary pane for the device subcategory, click New.
5. Complete all relevant information for the STM device, providing the IP address/range from Step 2 in the Access IP field of the Summary
6. Click Save when you’re done entering device information for the STM.
7. Follow the instructions in Creating a Report to add information about the STM device to a business service report, and then use the instructions in Adding Widgets to Dashboards to add it to your dashboard.

Related Links

Adding Devices to the CMDB Outside of Discovery

Creating CMDB Groups and Adding Objects to Them

Creating a Report

Adding Widgets to Dashboards

 

Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!

Creating Dynamic CMDB Group Policies

This setting allows you to write rules to put devices in CMDB Device Group and Business Service Groups of your choice. When a device is discovered, the policies defined here are applied and the device is assigned to the group(s) defined in the matching policies.

To create a new CMDB Group Policy

1. Go to Admin > General Settings > Discovery > CMDB Group
2. Click Add
3. For matching conditions – enter the following information
   1. Organization – the organization which this rule applies to
   2. Vendor – the matching device vendor – select from the list
   3. Model – the matching device model – select from the list
   4. Host Name – matching device host name via regular expression match
   5. IP Range – matching device access IP – format is single IP, IP range, CIDR
4. For Actions (Add To) – enter the following information
   1. Groups – specify the groups which the matching devices will be added to
   2. Biz Services – specify the business services which the matching devices will be added to

To apply one or more CMDB Group policies,

1. Select one or more policies and click Apply or Click Apply All to apply all policies.
2. Once a policy is saved, then next discovery will apply these policies. That means, discovered devices will belong to the groups and business services defined in the policies.

Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!

Decommissioning a device

Decommissioning a device lets you re-assign the IP address to a new device but still keep the old device in CMDB for historical purposes.

To decommission a device

1. Go to CMDB > Devices 2. Select the device.
2. Click on the menu under Name and select Decommission.
3. Provide a Reason and Select OK to decommission the device
4. Consequences of decommissioning
   1. Device will be moved to CMDB > Devices > Decommission folder
   2. Device will be removed from maintenance calendars
   3. Performance monitoring will stop
   4. A new device with the same IP can be discovered

To re-commission the device

1. Go to CMDB > Devices > Decommission 2. Select the device.

3. Click on the menu under Name and select Recommission.
4. The device will be moved back to the folder where it was when it was decommissioned. 5. Performance monitoring will resume

Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!