Troubleshooting and logging

This section explains how to troubleshoot logging configuration issues, as well as connection issues, that you may have with your FortiGate unit and a log device. This section also contains information about how to use log messages when troubleshooting issues that are about other FortiGate features, such as VPN tunnel errors.

Using log messages to help in troubleshooting issues

Log messages can help when troubleshooting issues that occur, since they can provide details about what is occurring. The uses and methods for involving logging in troubleshooting vary depending on the problem. The following are examples of how log messages can assist when troubleshooting networking issues.

Using IPS packet logging in diagnostics

This type of logging should only be enabled when you need to know about specific diagnostic information, for example, when you suspect a signature is triggered by a false positive. These log messages can help troubleshoot individual problems with misidentified or missing packets and network intrusions involving malicious packets.

To configure IPS packet logging

1. Go to Security Profiles > Intrusion Protection.
2. Select the IPS sensor that you want to enable IPS packet logging on, and then select Edit.
3. In the filter options, enable Packet Logging.
4. Select OK.

If you want to configure the packet quota, number of packets that are recorded before alerts and after attacks, use the following procedure.

To configure additional settings for IPS packet logging

1. Log in to the CLI.
2. Enter the following to start configuring additional settings:

config ips settings set ips-packet-quota <integer> set packet-log-history <integer> set packet-log-post-attack <integer>

end

Using HA log messages to determine system status

When the FortiGate unit is in HA mode, you may see the following log message content within the event log:

type=event subtype=ha level=critical msg= “HA slave heartbeat interface internal lost neighbor information”

OR

type=event subtype=ha level=critical msg= “Virtual cluster 1 of group 0 detected new joined HA member” OR

type=event subtype=ha level=critical msg= “HA master heartbeat interface internal get peer information”

The log messages occur within a given time, and indicate that the units within the cluster are not aware of each other anymore. These log messages provide the information you need to fix the problem.

Connection issues between FortiGate unit and logging devices

If external logging devices are not recording the log information properly or at all, the problem will likely be due to one of two situations: no data is being received because the log device cannot be reached, or no data is being sent because the FortiGate unit is no longer logging properly.

Unable to connect to a supported log device

After configuring logging to a supported log device, and testing the connection, you may find you cannot connect. To determine whether this is the problem:

1. Verify that the information you entered is correct; it could be a simple mistake within the IP address or you may have not selected Apply on the Log Settings page after changing them, which would prevent them from taking effect.
2. Use execute ping to see if you can ping to the log device.
3. If you are unable to ping to the log device, check to see if the log device itself working and that it is on the network and assigned an appropriate address.

FortiGate unit has stopped logging

If the FortiGate unit stopped logging to a device, test the connection between both the FortiGate unit and device using the execute ping command. The log device may have been turned off, is upgrading to a new firmware version, or just not working properly.

The FortiGate unit may also have a corrupted log database. When you log into the web-based manager and you see an SQL database error message, it is because the SQL database has become corrupted. View “SQL database errors” in the next section before taking any further actions, to avoid losing your current logs.

Log database issues

If attempting to troubleshoot issues with the SQL log database, use the following to help guide you to solving issues that occur.

SQL statement syntax errors

There may be errors or inconsistencies in the SQL used to maintain the database. Here are some example error messages and possible causes:

You have an error in your SQL syntax (remote/MySQL)

or

ERROR: syntax error at or near… (local/PostgreSQL)

• Verify that the SQL keywords are spelled correctly, and that the query is well-formed.
• Table and column names are demarked by grave accent (`) characters. Single (‘) and double (“) quotation marks will cause an error.

No data is covered.

• The query is correctly formed, but no data has been logged for the log type. Verify that you have configured the FortiGate unit to save that log type. On the Log Settings page, make sure that the log type is checked.

Connection problems

If well-formed SQL queries do not produce results, and logging is turned on for the log type, there may be a database configuration problem with the remote database.

Ensure that:

l MySQL is running and using the default port 3306. l You have created an empty database and a user who has read/write permissions for the database. l Here is an example of creating a new MySQL database named fazlogs, and adding a user for the database:

1. #Mysql –u root –p
2. mysql> Create database fazlogs;
3. mysql> Grant all privileges on fazlogs.* to ‘fazlogger’@’*’ identified by ‘fazpassword’;
4. mysql> Grant all privileges on fazlogs.* to ‘fazlogger’@’localhost’ identified by ‘fazpassword’;

SQL database errors

If the database seems inacessible, you may encounter the following error message after upgrading or downgrading the FortiGate unit’s firmware image.

Example of an SQL database error message

The error message indicates that the SQL database is corrupted and cannot be updated with the SQL schemas any more. When you see this error message, you can do one of the following:

l select Cancel and back up all log files; then select Rebuild to blank and rebuild the database. l select Rebuild immediately, which will blank the database and previous logs will be lost.

Until the database is rebuilt, no information will be logged by the FortiGate unit regardless of the log settings that are configured on the unit. When you select Rebuild, all logs are lost because the SQL database is erased and then rebuilt again. Logging resumes automatically according to your settings after the SQL database is rebuilt.

To view the status of the database, use the diagnose debug sqldb-error status command in the CLI. This command will inform you whether the database has errors present.

If you want to view the database’s errors, use the diagnose debug sqldb-error read command in the CLI. This command indicates exactly what errors occurred, and what tables contain those errors.

Log files are backed up using the execute backup {disk | memory } {alllogs | logs} command

in the CLI. You must use the text variable when backing up log files because the text variable allows you to view the log files outside the FortiGate unit. When you back up log files, you are really just copying the log files from the database to a specified location, such as a TFTP server.

Logging daemon (Miglogd)

The number of logging daemon child processes has been made available for editing. A higher number can affect performance, and a lower number can affect log processing time, although no logs will be dropped or lost if the number is decreased.

If you are suffering from performance issues, you can alter the number of logging daemon child processes, from 0 to 15, using the following syntax. The default is 8.

Troubleshooting and logging                                                                                          Logging daemon (Miglogd)

config system global set miglogd-children <integer> end

Advanced logging

This section explains how to configure other log features within your existing log configuration. You may want to include other log features after initially configuring the log topology because the network has either outgrown the initial configuration, or you want to add additional features that will help your network’s logging requirements.

The following topics are included in this section:

l Log backup and restore tools l Configuring logging to multiple Syslog servers l Using Automatic Discovery to connect to a FortiAnalyzer unit l Activating a FortiCloud account for logging purposes l Viewing log storage space l Customizing and filtering log messages l Viewing logs from the CLI l Configuring NAC Quarantine logging l Logging local-in policies l Tracking specific search phrases in reports l Interpreting and configuring FSSO syslog log messages

Log backup and restore tools

Local disk logs can now be backed up and restored to local files, using CLI commands:

execute log backup <filename> execute log restore <filename>

Restoring logs will wipe the current log and report content off the disk.

Logs can also now be exported to a USB storage device, as LZ4 compressed files, from both CLI and GUI. When you insert a USB drive into the FortiGate’s USB port, the USB menu will appear in the GUI. The menu shows the amount of storage on the USB disk, and the log file size, and you can select Copy to USB to copy the log data to the drive.

Logging and reporting for large networks

This section explains how to configure the FortiGate unit for logging and reporting in a larger network, such as an enterprise network. To set up this type of network, you are modifying the default log settings, and you are also modifying the default report.

The following procedures are examples and can be used to help you when configuring your own network’s log topology.

Since some of these settings must be modified or enabled or disabled in the CLI, it is recommended to review the FortiGate CLI Reference for any additional information about the commands used herein, as well as any that you would need to use in your own newtork’s log topology.

Modifying default log device settings

The default log device settings must be modified so that system performance is not compromised. The FortiGate unit, by default, has all logging of FortiGate features enabled and well as logging to either the FortiGate unit’s system memory or hard disk, depending on the model.

Modifying multiple FortiGate units’ system memory default settings

When the FortiGate unit’s default log device is its system memory, you can modify it to fit your log network topology. In this topic, the following is an example of how you can modify these default settings.

To modify the default system memory settings

1. Log in to the CLI.
2. Enter the following command syntax to modify the logging settings:

config log memory setting set status enable

end

3. Enter the following command syntax to modify the FortiGate features that are enabled for logging:

config log memory filter set forward-traffic enable set local-traffic enable set sniffer-traffic enable set anomaly enable set voip enable set multicast-traffic enable

set dns enable

end

4. Repeat steps 2 and 3 for the other FortiGate units.
5. Test the modified settings using the procedure below.

Modifying multiple FortiGate units’ hard disk default log settings

You will have to modify each FortiGate unit’s hard disk default log settings. The following is an example of how to modify these default settings.

To modify the default hard disk settings

1. Log in to the CLI.
2. Enter the following command syntax to modify the logging settings:

config log disk setting set ips-archive disable set status enable set max-log-file-size 1000 set storage Internal set log-quota 100 set report-quota 100

end

3. In the CLI, enter the following to disable certain event log messages that you do not want logged:

config log eventfilter set event enable set system enable set vpn enable set user enable set router disable set wan-opt disable

end

4. Repeat the steps 2 to 4 for the other FortiGate units.
5. Test the modified settings using the procedure below.

Testing the modified log settings

After modifying both the settings and the FortiGate features for logging, you can test that the modified settings are working properly. This test is done in the CLI.

To test sending logs to the log device

1. In the CLI, enter the following command syntax:

diag log test

When you enter the command, the following appears:

generating a system event message with level – warning generating an infected virus message with level – warning generating a blocked virus message with level – warning generating a URL block message with level – warning generating a DLP message with level – warning generating an IPS log message generating an anomaly log message generating an application control IM message with level – information generating an IPv6 application control IM message with level – information generating deep application control logs with level – information generating an antispam message with level – notification generating an allowed traffic message with level – notice generating a multicast traffic message with level – notice generating a ipv6 traffic message with level – notice generating a wanopt traffic log message with level – notification generating a HA event message with level – warning generating netscan log messages with level – notice generating a VOIP event message with level – information generating a DNS event message with level – information generating authentication event messages generating a Forticlient message with level – information generating a URL block message with level – warning

2. In the web-based interface, go to Log & Report > System Events, and view the logs to see some of the recently generated test log messages.

You will be able to tell the test log messages from real log messages because they do not have “real” information; for example, the test log messages for the vulnerability scan contain the destination IP address of 1.1.1.1 or 2.2.2.2.

Configuring the backup solution

Even though you are logging to multiple FortiAnalyzer units, this is more of a redundancy solution rather than a complete backup solution in this example.

The multiple FortiAnalyzer units act similar to a HA cluster, since if one FortiAnalyzer unit fails, the others continue storing the logs they receive. In a backup solution, the logs are backed up to another secure location if something happens to the log device.

A good alternate or redundant option is the FortiCloud service, which can provide secure online logging and management for multiple devices.

Configuring logging to multiple FortiAnalyzer units

The following example shows how to configure logging to multiple FortiAnalyzer units. Configuring multiple FortiAnalyzer units is quick and easy; however, you can only configure up to three FortiAnalyzer units per FortiGate unit.

To configure multiple FortiAnalyzer units

1. In the CLI, enter the following command syntax to configure the first FortiAnalyzer unit: config log fortianalyzer setting set status enable set server 172.20.120.22 set max-buffer-size 1000 set buffer-max-send 2000 set address-mode static set conn-timeout 100 set monitor-keepalive-period 120 set monitor-failure-retry-period 2000

end

2. Disable the features that you do not want logged, using the following example command syntax. You can view the CLI Reference to see what commands are available.

config log fortianalyzer filter set forward-traffic (enable | disable) … end

3. Enter the following commands for the second FortiAnalyzer unit: config log fortianalyzer2 setting set status enable set server 172.20.120.23 set max-buffer-size 1000 set buffer-max-send 2000 set address-mode static set conn-timeout 100 set monitor-keepalive-period 120 set monitor-failure-retry-period 2000

end

4. Disable the features that you do not want logged, using the following example command syntax.

config log fortianalyzer2 filter set event (enable | disable) … end

5. Enter the following commands for the last FortiAnalyzer unit: config log fortianalyzer3 setting set status enable set server 172.20.120.23 set max-buffer-size 1000 set buffer-max-send 2000 set address-mode static set conn-timeout 100 set monitor-keepalive-period 120 set monitor-failure-retry-period 2000

end

6. Disable the features that you do not want logged, using the following example command syntax.

config log fortianalyzer3 filter set voip (enable | disable) … end

7. Test the configuration by using the procedure, “Testing the modified log settings”.
8. On the other FortiGate units, configure steps 1 through 6, ensuring that logs are being sent to the FortiAnalyzer units.

Configuring logging to the FortiCloud server

The FortiCloud server can be used as a redundant backup, or your primary logging solution. The following assumes that this service has already been registered, and a subscription has been purchased for expanded space. The following is an example of how to these settings are configured for a network’s log configuration. You need to have access to both the CLI and the web-based manager when configuring uploading of logs. The upload time and interval settings can be configured in the web-based interface.

To configure logging to the FortiCloud server

1. Go to Dashboard and click Login next to FortiCloud in the License Information widget.
2. Enter your username and password, and click OK. (Or register, if you have not yet done so.)
3. Logs will automatically be uploaded to FortiCloud as long as your FortiGate is linked to your FortiCloud account.
4. To configure the upload time and interval, go to Log & Report > Log Settings.
5. Under the Remote Logging and Archiving header, you can select your desired upload time.
6. With FortiCloud you can easily store and access FortiGate logs that can give you valuable insight into the health and security of your network.

Logging and reporting for small networks

This section explains how to configure the FortiGate unit for logging and reporting in a small office or SOHO/SMB network. To properly configure this type of network, you will be modifying the default log settings, as well as the default FortiOS report.

The following procedures are examples and can be used to help you when configuring your own network’s log topology. Since some of these settings must be modified or enabled or disabled in the CLI, it is recommended to review the FortiGate CLI Reference for any additional information about the commands used herein, as well as any that you would need to use in your own network’s log topology.

Modifying default log device settings

The default log device settings must be modified so that system performance is not compromised. The FortiGate unit, by default, has all logging of FortiGate features enabled, except for traffic logging. The default logging location will be either the FortiGate unit’s system memory or hard disk, depending on the model. Units with a flash disk are not recommended for disk logging.

Modifying the FortiGate unit’s system memory default settings

When the FortiGate unit’s default log device is its system memory, the following is modified for a small network topology. The following is an example of how to modify these default settings.

To modify the default system memory settings

1. Log in to the CLI.
2. Enter the following command syntax to modify the logging settings:

config log memory setting set status enable

end

3. The following example command syntax modifies which FortiGate features that are enabled for logging:

config log memory filter set forward-traffic enable set local-traffic enable set sniffer-traffic enable set anomaly enable set voip disable set multicast-traffic enable

set dns enable

end

Modifying the FortiGate unit’s hard disk default settings

When the FortiGate unit’s default log device is its hard disk, you need to modify those settings to your network’s logging needs so that you can effectively log what you want logged. The following is an example of how to modify these default settings.

To modify the default hard disk settings

1. Log in to the CLI.
2. Enter the following command syntax to modify the logging settings:

config log disk setting set ips-archive disable set status enable set max-log-file-size 1000 set storage FLASH set log-quota 100 set report-quota 100

end

3. In the CLI, enter the following to disable certain event log messages that you do not want logged:

config log eventfilter set event enable set system enable set vpn disable set user enable set router disable set wan-opt disable

end

Testing sending logs to the log device

After modifying both the settings and the FortiGate features for logging, you can test that the modified settings are working properly. This test is done in the CLI.

To test sending logs to the log device

1. In the CLI, enter the following command syntax:

diag log test

When you enter the command, the following appears:

generating a system event message with level – warning generating an infected virus message with level – warning generating a blocked virus message with level – warning generating a URL block message with level – warning generating a DLP message with level – warning generating an IPS log message generating an anomaly log message generating an application control IM message with level – information generating an IPv6 application control IM message with level – information generating deep application control logs with level – information generating an antispam message with level – notification generating an allowed traffic message with level – notice generating a multicast traffic message with level – notice generating a ipv6 traffic message with level – notice generating a wanopt traffic log message with level – notification generating a HA event message with level – warning generating netscan log messages with level – notice generating a VOIP event message with level – information generating a DNS event message with level – information generating authentication event messages generating a Forticlient message with level – information generating a URL block message with level – warning

2. In the web-based interface, go to Log & Report > System Events, and view the logs to see some of the recently generated test log messages.

You will be able to tell the test log messages from real log messages because they do not have “real” information; for example, the test log messages for the vulnerability scan contain the destination IP address of 1.1.1.1 or 2.2.2.2.

Configuring the backup solution

A backup solution provides a way to ensure logs are not lost. The following backup solution explains logging to a FortiCloud server and uploading logs to a FortiAnalyzer unit. With this backup solution, there can be three simultaneous storage locations for logs, the first being the FortiGate unit itself, the FortiAnalyzer unit and then the FortiCloud server.

Configuring logging to a FortiCloud server

The FortiCloud server can be used as a redundant backup, or your primary logging solution. The following assumes that this service has already been registered, and a subscription has been purchased for expanded space. The following is an example of how to these settings are configured for a network’s log configuration. You need to have access to both the CLI and the web-based manager when configuring uploading of logs. The upload time and interval settings can be configured in the web-based interface.

To configure logging to the FortiCloud server

1. Go to Dashboard and click Login next to FortiCloud in the License Information widget. 2. Enter your username and password, and click OK. (Or register, if you have not yet done so.)
2. Logs will automatically be uploaded to FortiCloud as long as your FortiGate is linked to your FortiCloud account.
3. To configure the upload time and interval, go to Log & Report > Log Settings.
4. Under the Logging and Archiving header, you can select your desired upload time.

With FortiCloud you can easily store and access FortiGate logs that can give you valuable insight into the health and security of your network.

Configuring uploading logs to the FortiAnalyzer unit

The logs will be uploaded to the FortiAnalyzer unit at a scheduled time. The following is an example of how to upload logs to a FortiAnalyzer unit.

To upload logs to a FortiAnalyzer unit

1. Go to Log & Report > Log Settings.
2. In the Remote Logging and Archiving section, select the check box beside Send Logs to FortiAnalyzer/FortiManager.
3. Select FortiAnalyzer (Daily at 00:00).
4. Enter the FortiAnalyzer unit’s IP address in the IP Address
5. To configure the daily upload time, open the CLI.
6. Enter the following to configure when the upload occurs, and the time when the unit uploads the logs:

config log fortianalyzer setting set upload-interval {daily | weekly | monthly} set upload-time <hh:mm>

end

7. To change the upload time, in the web-based manager, select Change beside the upload time period, and then make the changes in the Upload Schedule window. Select OK.

Testing uploading logs to a FortiAnalyzer unit

You should test that the FortiGate unit can upload logs to the FortiAnalyzer unit, so that the settings are configured properly.

To test the FortiAnalyzer upload settings

1. Go to Log & Report > Log Settings.
2. In the Logging and Archiving section, under Send Logs to FortiAnalyzer/FortiManager, change the time to the current time by selecting Change.

For example, the current time is 11:10 am, so Change now has the time 11:10.

3. Select OK.

The logs will be immediately sent to the FortiAnalyzer unit, and will be available to view from within the FortiAnalyzer’s interface.

Best practices: Log management

When the FortiGate unit records FortiGate activity, valuable information is collected that provides insight into how to better protect network traffic against attacks, including misuse and abuse. There is a lot to consider before enabling logging on a FortiGate unit, such as what FortiGate activities to enable and which log device is best suited for your network’s logging needs. A plan can help you in deciding the FortiGate activities to log, a log device, as well as a backup solution in the event the log device fails. This plan should provide you with an outline, similar to the following:

l what FortiGate activities you want and/or need logged (for example, security features) l the logging device best suited for your network structure l if you want or require archiving of log files l ensuring logs are not lost in the event a failure occurs.

After the plan is implemented, you need to manage the logs and be prepared to expand on your log setup when the current logging requirements are outgrown. Good log management practices help you with these tasks.

Log management practices help you to improve and manage logging requirements. Logging is an ever-expanding tool that can seem to be a daunting task to manage. The following management practices will help you when issues arise, or your logging setup needs to be expanded.

1. Revisit your plan on a yearly basis to verify that your logging needs are being met by your current log setup. For example, your company or organization may require archival logging, but not at the beginning of your network’s lifespan. Archival logs are stored on a FortiGate unit’s local hard drive, a FortiAnalyzer unit, or a FortiCloud server, in increasing order of size.
2. Configure an alert message that will notify you of activities that are important to be aware about. For example: if a branch office does not have a FortiGate administrator, you will need to know at all times that the IPsec VPN tunnel is still up and running. An alert email notification message can be configured to send only if IPsec tunnel errors occur.
3. If your organization or company uses peer-to-peer programs such as Skype or other instant messaging software, use the Applications FortiView dashboard, or the Executive Summary’s report widget (Top 10 Application Bandwidth Usage Per Hour Summary) to help you monitor the usage of these types of instant messaging software. These widgets can help you in determining how these applications are being used, including if there is any misuse and abuse. Their information is taken from application log messages; however, application log messages should be viewed as well since they contain the most detailed information.
4. Ensure that your backup solution is up-to-date. If you have recently expanded your log setup, you should also review your backup solution. The backup solution provides a way to ensure that all logs are not lost in the event that the log device fails or issues arise with the log device itself.