How to check hardware connections

How to check hardware connections

If there is no traffic flowing from the FortiGate unit, it may be a hardware problem. To check hardware connections:

  • Ensure the network cables are properly plugged into the interfaces.
  • Ensure there are connection lights for the network cables on the unit.
  • Change the cable if the cable or its connector are damaged or you are unsure about the cable’s type or quality—such as straight through or crossover, or possibly exposed wires at the connector.
  • Connect the FortiGate unit to different hardware.
  • Ensure the link status is set to Up for the interface, (see Network > Interface > Status). The link status is based on the physical connection and cannot be set in FortiOS.

 

If any of these solve the problem, it was a hardware connection problem. You should still perform some basic software connectivity tests to ensure complete connectivity. It might also be that the interface is disabled, or has its Administrative Status set to Down.

 

To enable an interface – web-based manager

1. Using the web-based management interface, go to System > Network > Interface.

2. Select and edit the interface to enable, such as port1.

3. Find Administrative Status at the bottom of the screen, and select Up.

4. Select Apply.

 

To enable an interface – CLI

config system interface edit port1

set status up next

end


Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!

How to verify the contents of the routing table (in NAT mode)

How to verify the contents of the routing table (in NAT mode)

When you have some connectivity, or possibly none at all a good place to look for information is the routing table. The routing table is where all the currently used routes are stored for both static and dynamic protocols. If a route is in the routing table, it saves the time and resources of a lookup. If a route is not used for a while and a new route needs to be added, the oldest least used route is bumped if the routing table is full. This ensures the most recently used routes stay in the table. If your FortiGate unit is in Transparent mode, you are unable to perform this step.

If the FortiGate is running in NAT mode, verify that all desired routes are in the routing table: local subnets, default routes, specific static routes, and dynamic routing protocols.

To check the routing table in the web-based manager, use the Routing Monitor by going to Router > Monitor > Routing Monitor.

 

In the CLI, use the command get router info routing-table all. Sample output:

FGT# get router info routing-table all

Codes:

K – kernel, C – connected, S – static, R – RIP, B – BGP O – OSPF, IA – OSPF inter area

N1 – OSPF NSSA external type 1, N2 – OSPF NSSA external type 2

E1 – OSPF external type 1, E2 – OSPF external type 2

i – IS-IS, L1 – IS-IS level-1, L2 – IS-IS level-2, ia – IS-IS inter area

* – candidate default

S* 0.0.0.0/0 [10/0] via 172.20.120.2, wan1

C 10.31.101.0/24 is directly connected, internal

C 172.20.120.0/24 is directly connected, wan1


Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!

Troubleshooting Common questions

Common questions

The general troubleshooting tips include, and can help answer, the following questions:

 

How to check hardware connections

  • Are all the cables and interfaces connected properly?
  • Is the LED for the interface green?

 

How to check FortiOS network settings

  • If you are having problems connecting to the management interface, is your protocol enabled on the interface for administrative access?
  • Is there an IP address on the interface?

How to check CPU and memory resources

  • Is your CPU running at almost 100 percent usage?
  • Are you running low on memory?

 

How to check modem status

  • Is the modem connected?
  • Are there PPP issues?

 

How to run ping and traceroute

  • Are you experiencing complete packet loss?

 

How to check the logs

  • Do you need to identify a problem?

 

How to verify the contents of the routing table (in NAT mode)

  • Are there routes in the routing table for default and static routes?
  • Do all connected subnets have a route in the routing table?
  • Does a route wrongly have a higher priority than it should?

 

How to verify the correct route is being used

  • Has the traffic been routed correctly?

 

How to verify the correct firewall policy is being used

  • Is the correct firewall policy applied to the expected traffic?

 

How to check the bridging information in Transparent mode

  • Are you having problems in Transparent mode?

 

How to check number of sessions used by UTM proxy

  • Have you reached the maximum number of sessions for a protocol?
  • Are new sessions failing to start for a certain protocol?

 

How to examine the firewall session list

  • Are there active firewall sessions?

 

How to check wireless information

  • Is the wireless network functioning properly?

 

How to verify FortiGuard connectivity

  • Is the FortiGate unit communicating properly with FortiGuard?

 

How to perform a sniffer trace (CLI and Packet Capture)

  • Is traffic entering the FortiGate unit and does it arrive on the expected interface?
  • Is the ARP resolution correct for the next-hop destination?
  • Is the traffic exiting the FortiGate unit to the destination as expected?
  • Is the traffic being sent back to the originator?

 

How to debug the packet flow

  • Is the traffic entering or leaving the FortiGate unit as expected?

Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!

FortiGuard troubleshooting

FortiGuard troubleshooting

The FortiGuard service provides updates to Antivirus, Antispam, IPS, Webfiltering, and more. The FortiGuard Distribution System (FDS) involves a number of servers across the world that provide updates to your FortiGate unit. Problems can occur both with connection to FDS, and its configuration on your local FortiGate unit. Some of the more common troubleshooting methods are listed here including

  • Troubleshooting process for FortiGuard updates
  • FortiGuard server settings

 

Troubleshooting process for FortiGuard updates

The following process are the logical steps to take when troubleshooting FortiGuard update problems. This includes antivirus (AV), intrusion protection services (IPS), antispam (AS), and web filtering (WB).

1. Does the device have a valid licence that includes these services?

Each device requires a valid FortiGuard license to access updates for some or all of these services. You can verify the support contract status for your devices at the Fortinet Support website — https://support.fortinet.com/.

2. If the device is part of an HA cluster, do all members of the cluster have the same level of support?

As with the previous step, you can verify the support contract status for all the devices in your HA cluster at the

Fortinet Support website.

3. Have services been enabled on the device?

To see the FortiGuard information and status for a device, in the web-based manager go to System > Config > FortiGuard. On that page you can verify the status of each component, and if required enable each service. If there are problems, see the FortiGuard section of the FortiOS Handbook.

4. Is the device able to communicate with FortiGuard servers?

At System > Config > FortiGuard you can also attempt to update AV and IPS, or test the availability of WF and

AS default and alternate ports. If there are problems, see the FortiGuard section of the FortiOS Handbook.

5. Is there proper routing to reach the FortiGuard servers?

Ensure there is a static or dynamic route that enables your ForitGate unit to reach the FortiGuard servers. Usually a generic default route to the internet is enough, but you may need to verify this if your network is complex.

6. Are there issues with DNS?

An easy way to test this is to attempt a traceroute from behind the FortiGate unit to an external network using the

FQDN for a location. If the traceroute FQDN name does not resolve, you have general DNS problems.

7. Is there anything upstream that might be blocking FortiGuard traffic, either on the network or ISP

side?

Many firewalls block all ports by default, and often ISPs block ports that are low. There may be a firewall between the FortiGate unit and the FortiGuard servers that is blocking the traffic. FortiGuard uses port 53 by default, so if it is being blocked you need to either open a hole for it, or change the port it is using.

8. Is there an issue with source ports?

It is possible that ports used to contact FortiGuard are being changed before reaching FortiGuard or on the return trip before reaching your FortiGate unit. A possible solution for this is to use a fixed-port at NATd firewalls to ensure the port remains the same. Packet sniffing can be used to find more information on what is happening with ports.

9. Are there security policies that include antivirus?

If no security policies include antivirus, the antivirus databse will not be updated. If antivirus is included, only the database type used will be updated.

 

FortiGuard server settings

Your local FortiGate unit connects to remote FortiGuard servers get updates to FortiGuard information such as new viruses that may have been found or other new threats. This section demonstrates ways to display information about FortiGuard server information on your FortiGate unit, and how to use that information and update it to fix potential problems.

 

Displaying the server list

The get webfilter status or diag debug rating command shows the list of FDS servers the FortiGate unit is using to send web filtering requests. Rating requests are only sent to the server on the top of the list in normal operation. Each server is probed for Round Trip Time (RTT) every two minutes.

You can optionally add a refresh rate to the end of this command and that will determine how often the server list will be refreshed.

Rating may not be enabled on your FortiGate unit.

get webfilter status

Sample Output:

Locale : english

License : Contract

Expiration : Thu Oct 9 02:00:00 2011

-=- Server List (Mon Feb 18 12:55:48 2008) -=-

 

IP Weight RTT Flags TZ Packets CurrLost TotalLost
a.b.c.d 0 1 DI 2 1926879 0 11176
10.1.101.1 10 329   1 10263 0 633
10.2.102.2 20 169   0 16105 0 80
10.3.103.3 20 182   0 6741 0 776
10.4.104.4 20 184   0 5249 0 987
10.5.105.5 25 181   0 12072 0 178

 

Output Details

The Server List includes the IP addresses of alternate servers if the first entry cannot be reached. In this example the IP addresses are not public addresses

 

The following flags in get webfilter status indicate the server status:

  • D – the server was found through the DNS lookup of the hostname. If the hostname returns more than one IP address, all of them will be flagged with D and will be used first for INIT requests before falling back to the other servers.
  • I – the server to which the last INIT request was sent.
  • F – the server has not responded to requests and is considered to have failed.
  • T – the server is currently being timed.
  • S – means that rating requests can be sent to the server. The flag is set for a server only in two cases:

1. The server exists in the servers list received from the Fortimanager or any other INIT server.

2. The servers list received from the FortiManager is empty so the Fortimanager itself would be the only server known by Fortigate thus it should be used as the rating server.

The servers that are not currently serving will be pushed down to the bottom list (under the available serving servers, and on top of the failed servers) in order for the load-balance-servers feature in the config system fortiguard to work properly.

 

Sorting the server list

The server list is sorted first by weight. The server with the smallest RTT is put at the top of the list, regardless of weight. When a packet is lost (there has been no response in 2 seconds), it will be resent to the next server in the list. Therefore, the top position in the list is selected based on RTT while the other list positions are based on weight.

 

Calculating weight

The weight for each server increases with failed packets and decreases with successful packets. To lower the possibility of using a remote server, the weight is not allowed to dip below a base weight, calculated as the difference in hours between the FortiGate unit and the server times 10. The further away the server is, the higher its base weight and the lower in the list it will appear.


Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!

FortiAnalyzer/FortiManager ports

FortiAnalyzer/FortiManager ports

If you have a FortiAnalyzer unit or FortiManager unit on your network you may need to use the following ports for troubleshooting network traffic.

 

Functionality                            Port(s)

DNS lookup                                 UDP 53

NTP synchronization                   UDP 123

Windows share                            UDP 137-138

SNMP traps                                 UDP 162

Syslog, log forwarding                 UDP 514

Log and report upload                 TCP 21 or TCP 22

SMTP alert email                         TCP 25

User name LDAP queries for reports    TCP 389 or TCP 636

RVS update                                 TCP 443

RADIUS authentication               TCP 1812

Log aggregation client                 TCP 3000


Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!

FortiOS ports

FortiOS ports

In the TCP and UDP stacks, there are 65 535 ports available for applications to use when communicating with each other. Many of these ports are commonly known to be associated with specific applications or protocols. These known ports can be useful when troubleshooting your network.

 

Use the following ports while troubleshooting the FortiGate device:

 

Port(s)                                      Functionality

UDP 53                                        DNS lookup, RBL lookup

UDP 53 or UDP 8888                   FortiGuard Antispam or Web Filtering rating lookup

UDP 53 (default) or UDP 8888 and UDP 1027 or UDP 1031

FDN Server List – source and destination port numbers vary by originating or reply traffic. See the article “How do I troubleshoot performance issues when FortiGuard Web Filtering is enabled?” in the Knowledge Base.

 

UDP 123                                      NTP Synchronization

UDP 162                                      SNMP Traps

UDP 514

SYSLOG – All FortiOS versions can use syslog to send log messages to remote syslog servers. FortiOS v2.80 and v3.0 can also view logs stored remotely on a FortiAnalyzer unit.

TCP 22                                        Configuration backup to FortiManager unit or FortiGuard Analysis and Man- agement Service.

TCP 25                                        SMTP alert email, encrypted virus sample auto-submit

TCP 389 or TCP 636                   LDAP or PKI authentication

TCP 443

FortiGuard Antivirus or IPS update – When requesting updates from a FortiManager unit instead of directly from the FDN, this port must be recon- figured as TCP 8890.

TCP 443                                      FortiGuard Analysis and Management Service

TCP 514                                      FortiGuard Analysis and Management Service log transmission (OFTP)

 

Port(s)                                      Functionality

TCP 541                                      SSL Management Tunnel to FortiGuard Analysis and Management Service

(FortiOS v3.0 MR6 or later)

TCP 514                                      Quarantine, remote access to logs and reports on a FortiAnalyzer unit, device registration with FortiAnalyzer units (OFTP)

TCP 1812                                    RADIUS authentication

TCP 8000 and TCP 8002             FSSO

TCP 10151                                  FortiGuard Analysis and Management Service contract validation


Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!

Troubleshooting tools

Troubleshooting tools

FortiOS provides a number of tools that help with troubleshooting both hardware and software issues. These tools include diagnostics and ports; ports are used when you need to understand the traffic coming in or going out on a specific port, for example, UDP 53, which is used by the FortiGate unit for DNS lookup and RBL lookup.

 

This section also contains information about troubleshooting FortiGuard issues. This section contains the following topics:

  • FortiOS diagnostics
  • FortiOS ports
  • FortiAnalyzer/FortiManager ports
  • FortiGuard troubleshooting

 

FortiOS diagnostics

A collection of diagnostic commands are available in FortiOS for troubleshooting and performance monitoring. Within the CLI commands, the two main groups of diagnostic commands are get and diagnose commands. Both commands display information about system resources, connections, and settings that enable you to locate and fix problems, or to monitor system performance.

 

This topic includes diagnostics commands to help with:

  • Check date and time
  • Resource usage
  • Proxy operation
  • Hardware NIC
  • Traffic trace
  • Session table
  • Firewall session setup rate
  • Finding object dependencies
  • Flow trace
  • Packet sniffing and packet capture
  • NPU based interfaces
  • Debug command
  • The execute tac report command
  • Other commands

 

Check date and time

The system date and time are important for FortiGuard services, when logging events, and when sending alerts. The wrong time will make the log entries confusing and difficult to use.

Use Network Time Protocol (NTP) to set the date and time if possible. This is an automatic method that does not require manual intervention. However, you must ensure the port is allowed through the firewalls on your network. FortiToken synchronization requires NTP in many situations.

 

How to check the date and time – web-based manager

1. Go to System Information > System Time on the dashboard.

Alternately, you can check the date and time using the CLI commands execute date and execute time.

2. If required, select Change to adjust the date and time settings.

You can set the time zone, date and time, and select NTP usage. In the CLI, use the following commands to change the date and time:

config system global

set timezone (use ? to get a list of IDs and descriptions of their timezone)

end

config system ntp set type custom

config ntpserver edit 1

set server “ntp1.fortinet.net”

next edit 2

set server “ntp2.fortinet.net”

next

end

set ntpsync enable set syncinterval 60 end

 

Resource usage

Each program running on a computer has one or more processes associated with it. For example if you open a Telnet program, it will have an associated telnet process. The same is true in FortiOS. All the processes have to share the system resources in FortiOS including memory and CPU.

Use get system performance status command to show the FortiOS performance status. Sample output:

FGT#get system performance status

CPU states: 0% user 0% system 0% nice 100% idle CPU0 states: 0% user 0% system 0% nice 100% idle CPU1 states: 0% user 0% system 0% nice 100% idle CPU2 states: 0% user 0% system 0% nice 100% idle CPU3 states: 0% user 0% system 0% nice 100% idle Memory states: 25% used

Average network usage: 0 kbps in 1 minute, 0 kbps in 10 minutes, 0 kbps in 30 minutes

Average sessions: 5 sessions in 1 minute, 5 sessions in 10 minutes, 4 sessions in 30 minutes

Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes

Virus caught: 0 total in 1 minute

IPS attacks blocked: 0 total in 1 minute

Uptime: 0 days, 12 hours, 7 minutes

 

Monitor the CPU/memory usage of internal processes using the following command:

 

get system performance top <delay> <max_lines>

 

The data listed by the command includes the name of the daemon, the process ID, whether the process is sleeping or running, the CPU percentage being used, and the memory percentage being used.

 

Sample output:

FGT#get system performance top 10 100

Run Time: 0 days, 11 hours and 30 minutes

0U, 0S, 100I; 1977T, 1470F, 121KF

 

pyfcgid 120 S   0.0 1.3
pyfcgid 121 S   0.0 1.3
pyfcgid 122 S   0.0 1.3
pyfcgid 53 S   0.0 1.3
ipsengine 75 S < 0.0 1.3
ipsengine 66 S < 0.0 1.3
ipsengine 73 S < 0.0 1.3
ipsengine 74 S < 0.0 1.3
ipsengine 79 S < 0.0 1.3
ipsengine 80 S < 0.0 1.3
cmdbsvr 43 S   0.0 1.0
proxyworker 110 S   0.0 1.0
proxyworker 111 S   0.0 1.0
httpsd 125 S   0.0 0.8
httpsd 52 S   0.0 0.8
httpsd 124 S   0.0 0.8
newcli 141 R   0.0 0.7
newcli 128 S   0.0 0.7
fgfmd 102 S   0.0 0.7
iked 86 S   0.0 0.7

 

Proxy operation

Monitor proxy operations using the following command:

diag test application <application> <option>

The <application> value can include the following:

 

http                    HTTP proxy.

 

smtp                    SMTP proxy.

 

ftpd                    FTP proxy.

 

pop3                    POP3 proxy.

 

imap                    IMAP proxy.

 

nntp                    NNTP proxy.

 

proxyacceptor          Proxy acceptor.

 

proxyworker            Proxy worker.

 

scanunit               Scanning unit.

 

sslacceptor            SSL proxy.

 

sslworker              SSL proxy.

 

ssh                     SSH proxy.

 

harelay                 HA relay daemon.

 

hasync                  HA sync daemon.

 

forticldd              FortiCloud daemon.

 

miglogd                 Miglog logging daemon.

 

urlfilter              URL filter daemon.

 

ovrd                    Override daemon.

 

ipsmonitor             ips monitor

 

ipsengine              ips sensor

 

ipldbd                  IP load balancing daemon.

 

ddnscd                  DDNS client daemon.

 

snmpd                   SNMP daemon.

 

acd                     Aggregate Controller.

 

dnsproxy               DNS proxy.

 

sflowd                  sFlow daemon.

 

init                    init process.

 

l2tpcd                  L2TP client daemon.

 

dhcprelay              DHCP relay daemon.

 

pptpcd                  PPTP client.

 

wccpd                   WCCP daemon.

 

wad                     WAD related processes.

 

radiusd                 RADIUS daemon.

 

sqldb                   SQL database daemon.

 

reportd                 Report daemon.

 

dlpfingerprint         DLP fingerprint daemon.

 

dlpfpcache             DLP fingerprint cache daemon.

 

wpad                    WPA daemon.

 

fsd                     FortiExplorer daemon.

 

ipsufd                  IPS urlfilter daemon.

 

stp                     Spanning Tree Protocol daemon.

 

lted                    USB LTE daemon.

 

swctrl_authd           Switch controller authentication daemon.

 

forticron              Forticron daemon.

 

uploadd                 Upload daemon.

 

quarantined            Quarantine daemon.

 

dhcp6c                  DHCP6 client daemon.

 

info-sslvpnd           SSL-VPN info daemon.

 

thmd                    Traffic history monitor daemon.

 

dsd                     DLP Statistics daemon.

 

lnkmtd                  Link monitor daemon.

 

dhcp6r                  DHCP6 relay daemon.

 

fnbamd                  Fortigate non-blocking auth daemon.

 

 

The <option> value depends from the application value used in the command. Here are some examples:

  • If the application is http, the CLI command will be:

diag test application http <option>

 

The <option> value can be one from the following:

2                       Drop all connections.

22                      Drop max idle connections.

222                     Drop all idle connections.

4                       Display connection stat.

44                      Display info per connection.

444                     Display connections per state.

4444                    Display per-VDOM statistics.

44444                   Display information about idle connections.

55                      Display tcp info per connection.

6                       Display ICAP information.

70                      Disable ICAP ‘Allow: 204’ (default).

71                      Enable ICAP ‘Allow: 204’ .

72                      Drop all ICAP server connections.

11                      Display the SSL session ID cache statistics.

12                      Clear the SSL session ID cache statistics.

13                      Display the SSL session ID cache.

14                      Clear the SSL session ID cache.

80                      Show Fortinet bar SSL-VPN bookmark info.

81                      Show Fortinet bar SSL-VPN bookmark cache.

82                      Show Fortinet bar SSL-VPN bookmark LRU list.

  • If the application is ipsmonitor, the CLI command will be

diag test application ipsmonitor <option>

The <option> value can be one from the following:

1                       Display IPS engine information

2                       Toggle IPS engine enable/disable status

3                       Display restart log

4                       Clear restart log

5                       Toggle bypass status

6                       Submit attack characteristics now

10                      IPS queue length

11                      Clear IPS queue length

12                      IPS L7 socket statistics

13                      IPS session list

14                      IPS NTurbo statistics

15                      IPSA statistics

97                      Start all IPS engines

98                      Stop all IPS engines

99                      Restart all IPS engines and monitor

 

Hardware NIC

Monitor hardware network operations using the following command:

diag hardware deviceinfo nic <interface>

The information displayed by this command is important as errors at the interface are indicative of data link or physical layer issues which may impact the performance of the FortiGate unit.

The following is sample output when <interface> = internal:

System_Device_Name port5

Current_HWaddr 00:09:0f:68:35:60

Permanent_HWaddr 00:09:0f:68:35:60

Link up

Speed 100

Duplex full [……] Rx_Packets=5685708

Tx_Packets=4107073

Rx_Bytes=617908014

Tx_Bytes=1269751248

Rx_Errors=0

Tx_Errors=0

Rx_Dropped=0

Tx_Dropped=0 […..]

 

The diag hardware deviceinfo nic command displays a list of hardware related error names and values. The following table explains the items in the list and their meanings.

 

Possible hardware errors and meanings

Field                                         Definition

Rx_Errors = rx error count            Bad frame was marked as error by PHY.

Rx_CRC_Errors + Rx_Length_Errors – Rx_Align_Errors

This error is only valid in 10/100M mode.

Rx_Dropped or

Rx_No_Buffer_Count

Running out of buffer space.

Rx_Missed_Errors                       Equals Rx_FIFO_Errors + CEXTERR (Carrier Extension Error Count). Only valid in 1000M mode, whichis marked by PHY.

Tx_Errors = Tx_Aborted_ Errors

ECOL (Excessive Collisions Count). Only valid in half-duplex mode.

Tx_Window_Errors

LATECOL (Late Collisions Count). Late collisions are collisions that occur after 64-byte time into the transmission of the packet while working in 10 to100Mb/s data rate and 512-byte timeinto the transmission of the packet while working in the 1000Mb/s data rate. This register only increments if transmits are enabled and the device is in half-duplex mode.

Rx_Dropped                                See Rx_Errors.

 

Tx_Dropped                                 Not defined.

 

Collisions                                     Total number of collisions experienced by the transmitter. Valid in half- duplex mode.

 

Rx_Length_Errors                       Transmission length error.

 

Rx_Over_Errors                           Not defined.

 

Rx_CRC_Errors                           Frame CRC error.

 

Rx_Frame_Errors                        Same as Rx_Align_Errors. This error is only valid in 10/100M mode.

 

Field                                         Definition

Rx_FIFO_Errors                          Same as Rx_Missed_Errors – a missed packet count.

 

Tx_Aborted_Errors                      See Tx_Errors.

 

Tx_Carrier_Errors

The PHY should assert the internal carrier sense signal during every trans- mission. Failure to do so may indicate that the link has failed or the PHY has an incorrect link configuration. This register only increments if trans- mits are enabled. This register is not valid in internal SerDes 1 mode (TBI mode for the 82544GC/EI) and is only valid when the Ethernet controller is operating at full duplex.

 

Tx_FIFO_Errors                          Not defined.

 

Tx_Heartbeat_Errors                   Not defined.

 

Tx_Window_Errors                      See LATECOL.

 

Tx_Single_Collision_Frames

Counts the number of times that a successfully transmitted packed encountered a single collision. The value only increments if transmits are enabled and the Ethernet controller is in half-duplex mode.

 

Tx_Multiple_Collision_Frames   A Multiple Collision Count which counts the number of times that a trans- mit encountered more than one collision but less than 16. The value only increments if transmits are enabled and the Ethernet controller is in half- duplex mode.

Tx_Deferred

Counts defer events. A defer event occurs when the transmitter cannot immediately send a packet due to the medium being busy because another device is transmitting, the IPG timer has not expired, half-duplex deferral events are occurring, XOFF frames are being received, or the link is not up. This register only increments if transmits are enabled. This counter does not increment for streaming transmits that are deferred due to TX IPG.

 

Rx_Frame_Too_Longs               The Rx frame is over size.

 

Rx_Frame_Too_Shorts               The Rx frame is too short.

 

Rx_Align_Errors                          This error is only valid in 10/100M mode.

 

 

Symbol Error Count

Counts the number of symbol errors between reads – SYMERRS. The count increases for every bad symbol received, whether or not a packet is currently being received and whether or not the link is up. This register only increments in internal SerDes mode.

 

Traffic trace

Traffic tracing allows a specific packet stream to be followed. This is useful to confirm packets are taking the route you expected on your network.

View the characteristics of a traffic session though specific security policies using:

diag sys session

 

Trace per-packet operations for flow tracing using:

diag debug flow

 

Trace per-Ethernet frame using:

diag sniffer packet

 

Session table

A session is a communication channel between two devices or applications across the network. Sessions enable FortiOS to inspect and act on a sequential group of packets in a session all together instead of inspecting each packet individually. Each of these sessions has an entry in the session table that includes important information about the session.

 

Use as a tool

Session tables are useful troubleshooting tools because they allow you to verify connections that you expect to see open. For example, if you have a web browser open to browse the Fortinet website, you would expect a session entry from your computer, on port 80, to the IP for the Fortinet website. Another troubleshooting method is if there are too many sessions for FortiOS to process, you can examine the session table for evidence why this is happening.

The FortiGate session table can be viewed from either the CLI or the web-based manager. The most useful troubleshooting data comes from the CLI. The session table in web-based manager also provides some useful summary information, particularly the current policy number that the session is using.

 

Webbased manager session information

In the web-based manager you can view session information in the FortiView page. Sessions are categorized by Sources, Applications, Destinations, and All Sessions.

 

How to find which security policy a specific connection is using

Every program and device on your network must have a communication channel, or session, open to pass information. The FortiGate unit manages these sessions with its many features from traffic shaping, to antivirus scanning, and even blocking known bad web sites. Each session has an entry in the session table.

You may want to find information for a specific session, say a secure web browser session, for troubleshooting. For example if that web browser session is not working properly, you can check the session table to ensure the session is still active, and that it is going to the proper address. It can also tell you the security policy number it matches, so you can check what is happening in that policy.

 

1. Know your connection information.

You need to be able to identify the session you want. For this you need the source IP address (usually your computer), the destination IP address if you have it, and the port number which is determined by the program being used. Some commons ports are:

 

  • port 80 (HTTP for web browsing),
  • port 22 (SSH used for secure login and file transfers)
  • port 23 (telnet for a text connection)
  • port 443 (HTTPS for secure web browsing

2. Find your session and policy ID.

Follow System > FortiView> All Sessions. Find your session by finding your source IP address, destination IP address if you have it, and port number. The policy ID is listed after the destination information. If the list of sessions is very long, you can filter the list to make it easier to find your session.

3. When there are many sessions, use a filter to help you find your session.

If there are multiple pages of sessions it is difficult to find a single session. To help you in your search you can use a filter to block out sessions that you don’t want. Click the search icon on the column heading to select the filter. Select Source IP and enter your source IP address. Now only sessions that originate from your IP address will be displayed in the session table. If the list is still too long, you can do the same for the Source port. That will make it easy to find your session and the security policy ID. When you are finished remember to clear the filters.

 

CLI session information

The session table output from the CLI (diag sys session list) is very verbose. Even on a system with a small amount of traffic, displaying the session table will generate a large amount of output. For this reason, filters are used to display only the session data of interest.

You can filter a column in the web-based manager by clicking the search icon on the column heading or from the CLI by creating a filter.

An entry is placed in the session table for each traffic session passing through a security policy. The following command will list the information for a session in the table:

diag sys session list

 

Sample Output:

FGT# diag sys session list

session info: proto=6 proto_state=05 expire=89 timeout=3600 flags=00000000 av_idx=0 use=3 bandwidth=204800/sec guaranteed_bandwidth=102400/sec traffic=332/sec prio=0

logtype=session ha_id=0 hakey=4450

tunnel=/

state=log shape may_dirty

statistic(bytes/packets/err): org=3408/38/0 reply=3888/31/0 tuples=2

orgin->sink: org pre->post, reply pre->post oif=3/5 gwy=192.168.11.254/10.0.5.100 hook=post dir=org act=snat 10.0.5.100:1251->192.168.11.254:22(192.168.11.105:1251) hook=pre dir=reply act=dnat 192.168.11.254:22->192.168.11.105:1251(10.0.5.100:1251) pos/(before,after) 0/(0,0), 0/(0,0)

misc=0 domain_info=0 auth_info=0 ftgd_info=0 ids=0x0 vd=0 serial=00007c33 tos=ff/ff

 

Since output can be verbose, the filter option allows specific information to be displayed, for example:

diag sys session filter <option>

The <option> values available include the following:

clear                   Clear session filter.

 

dintf                   Destination interface.

 

dport                   Destination port.

 

dst                     Destination IP address.

 

duration               duration

 

expire                  expire

 

negate                  Inverse filter.

 

nport                   NAT’d source port

 

nsrc                    NAT’d source ip address

 

policy                  Policy ID.

 

proto                   Protocol number.

 

proto-state            Protocol state.

 

sintf                   Source interface.

 

sport                   Source port.

 

src                     Source IP address.

 

vd                      Index of virtual domain. -1 matches all.

 

 

Even though UDP is a sessionless protocol, the FortiGate unit still keeps track of the following two different states:

  • UDP reply not seen with a value of 0
  • UDP reply seen with a value of 1

 

The following illustrates FW session states from the session table:

 

State                        Meaning

 

log                              Session is being logged.

 

local                           Session is originated from or destined for local stack.

 

ext                              Session is created by a firewall session helper.

 

State                        Meaning

 

may_dirty

Session is created by a policy. For example, the session for ftp control chan- nel will have this state but ftp data channel will not. This is also seen when NAT is enabled.

 

ndr                             Session will be checked by IPS signature.

 

nds                             Session will be checked by IPS anomaly.

 

br                               Session is being bridged (TP) mode.

 

 

Firewall session setup rate

The number of sessions that can be established in a set period of time is useful information. A session is an end- to-end TCP/IP connection for communication with a limited lifespan. If you record the setup rate during normal operation, when you experience problems you have that setup rate with the current number to see if its very different. While this will not solve your problems, it can be a useful step to help you define your problem.

A reduced firewall session setup rate could be the result of a number of things from a lack of system resources on the FortiGate unit, to reaching the limit of your session count for your VDOM.

 

To view your session setup rate method 1- CLI

FGT# get sys performance status

CPU states: 0% user 0% system 0% nice 100% idle

Memory states: 10% used

Average network usage: 0 kbps in 1 minute, 0 kbps in 10 minutes,

13 kbps in 30 minutes

Average sessions: 31 sessions in 1 minute, 30 sessions in 10 minutes, 31 sessions in 30 minutes

Average session setup rate: 0.5 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes

Virus caught: 0 total in 1 minute

IPS attacks blocked: 0 total in 1 minute

Uptime: 44 days, 18 hours, 42 minutes

 

The information you are looking for is the Average sessions section, highlighted in the above output. In this example you can see there were 31 sessions in 1 minute, or an average of 0.5 sessions per second. The values for 10 minutes and 30 minutes allow you to take a longer average for a more reliable value if your FortiGate unit is working at maximum capacity. The smallest FortiGate unit can have 1 000 sessions established per second across the unit.

Remember that session setup rate is a global command. If you have multiple VDOMs configured with many sessions in each one, the session setup rate per VDOM will be slower than if there were no VDOMs configured.

 

Finding object dependencies

An administrator may not be permitted to delete a configuration object if there are other configuration objects that depend on it. This command identifies other objects which depend on or make reference to the configuration object in question. If an error is displayed that an object is in use and cannot be deleted, this command can help identify the source of the problem.

Another use is if you have a virtual interface with objects that depend on it, you need to find and remove those dependencies before you delete that interface.

 

CLI method

When running multiple VDOMs, this command is run in the Global configuration only and it searches for the named object both in the Global and VDOM configuration most recently used:

diag sys checkused <path.object.mkey>

 

For example, to verify which objects are referred to in a security policy with an ID of 1, enter the command as follows:

diag sys checkused firewall.policy.policyid 1

 

To check what is referred to by interface port1, enter the following command:

diag sys checkused system.interface.name port1

 

To show all the dependencies for an interface, enter the command as follows:

diag sys checkused system.interface.name <interface name>

 

Sample Output:

entry used by table firewall.address:name ‘10.98.23.23_host’

entry used by table firewall.address:name ‘NAS’ entry used by table firewall.address:name ‘all’

entry used by table firewall.address:name ‘fortinet.com’

entry used by table firewall.vip:name ‘TORRENT_10.0.0.70:6883′ entry used by table firewall.policy:policyid ’21’

entry used by table firewall.policy:policyid ’14’ entry used by table firewall.policy:policyid ’19’

In this example, the interface has dependent objects, including four address objects, one VIP, and three security policies.

 

Webbased manager method

In the web-based manager, the object dependencies for an interface can be easily checked and removed.

 

 

To remove interface object dependencies – web-based manager

1. Go to System > Interfaces.

The number in the Ref. column is the number of objects that refer to this interface.

2. Select the number in the Ref. column for the desired interface.

A Window listing the dependencies will appear.

3. Use these detailed entries to locate and remove object references to this interface.

The trash can icon will change from gray when all object dependencies have been removed.

4. Remove the interface by selecting the check box for the interface, and select Delete.

 

Flow trace

To trace the flow of packets through the FortiGate unit, use the following command:

diag debug flow trace start

 

If your network is using IPv4, follow packet flow by setting a flow filter using this command:

diag debug flow filter <option>

 

Filtering options include the following:

addr IPv4 address

clear clear filter

daddr destination IPv4 address

dport destination port negate inverse IPv4 filter port port

proto protocol number saddr source IPv4 address sport source port

vd index of virtual domain, -1 matches all

 

If your network is using IPv6, follow packet flow by setting a flow filter using this command:

diag debug flow filter6 <option>

 

Filtering options include the following:

addr IPv6 address

clear clear filter

daddr destination IPv6 address

dport destination port negate inverse IPv6 filter port port

proto protocol number saddr source IPv6 address sport source port

vd index of virtual domain, -1 matches all

 

Enable the output to be displayed to the CLI console using the following command:

diag debug flow show console enable

 

diag debug flow output is recorded as event log messages and are sent to a FortiCloud or a FortiAnalyzer unit if connected. Do not let this command run longer than necessary since it generates significant amounts of data.

 

Start flow monitoring with a specific number of packets using this command:

diag debug flow trace start <N>

 

Stop flow tracing at any time using:

diag debug flow trace stop

 

The following is an example of the flow trace for the device at the following IP address: 203.160.224.97

diag debug enable

diag debug flow filter addr 203.160.224.97 diag debug flow show console enable

diag debug flow show function-name enable diag debug flow trace start 100

 

Flow trace output example – HTTP

Connect to the web site at the following address to observe the debug flow trace. The display may vary slightly:

http://www.fortinet.com

 

Comment: SYN packet received:

 

id=20085 trace_id=209 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6,

192.168.3.221:1487->203.160.224.97:80) from port5.”

 

SYN sent and a new session is allocated:

 

id=20085 trace_id=209 func=resolve_ip_tuple line=2799 msg=”allocate a new session-00000e90″

 

Lookup for next-hop gateway address:

 

id=20085 trace_id=209 func=vf_ip4_route_input line=1543 msg=”find a route: gw-192.168.11.254 via port6″

 

Source NAT, lookup next available port:

 

id=20085 trace_id=209 func=get_new_addr line=1219 msg=”find SNAT: IP-192.168.11.59, port-31925″ direction“

 

Matched security policy. Check to see which policy this session matches:

 

id=20085 trace_id=209 func=fw_forward_handler line=317 msg=”Allowed by Policy-3: SNAT”

 

Apply source NAT:

 

id=20085 trace_id=209 func=  ip_session_run_tuple line=1502 msg=”SNAT 192.168.3.221->192.168.11.59:31925″

 

SYN ACK received:

 

id=20085 trace_id=210 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6, 203.160.224.97:80-

>192.168.11.59:31925) from port6.”

 

Found existing session ID. Identified as the reply direction:

 

id=20085 trace_id=210 func=resolve_ip_tuple_fast line=2727 msg=”Find an existing session, id-00000e90, reply direction”

 

Apply destination NAT to inverse source NAT action:

id=20085 trace_id=210 func=  ip_session_run_tuple line=1516 msg=”DNAT 192.168.11.59:31925->192.168.3.221:1487″

 

Lookup for next-hop gateway address for reply traffic:

id=20085 trace_id=210 func=vf_ip4_route_input line=1543 msg=”find a route: gw-192.168.3.221 via port5″

 

ACK received:

id=20085 trace_id=211 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6,192.168.3.221:1487->203.160.224.97:80) from port5.”

 

Match existing session in the original direction:

id=20085 trace_id=211 func=resolve_ip_tuple_fast line=2727 msg=”Find an existing session, id-00000e90, original direction”

 

Apply source NAT:

id=20085 trace_id=211 func=  ip_session_run_tuple line=1502 msg=”SNAT 192.168.3.221->192.168.11.59:31925″

 

Receive data from client:

id=20085 trace_id=212 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6, 192.168.3.221:1487->203.160.224.97:80) from port5.”

 

Match existing session in the original direction:

id=20085 trace_id=212 func=resolve_ip_tuple_fast line=2727 msg=”Find an existing session, id-00000e90, original direction”

 

Apply source NAT:

id=20085 trace_id=212 func=  ip_session_run_tuple line=1502 msg=”SNAT 192.168.3.221->192.168.11.59:31925″

 

Receive data from server:

id=20085 trace_id=213 func=resolve_ip_tuple_fast line=2700 msg=”vd-root received a packet(proto=6,

203.160.224.97:80->192.168.11.59:31925) from port6.”

 

Match existing session in reply direction:

id=20085 trace_id=213 func=resolve_ip_tuple_fast line=2727 msg=”Find an existing session, id-00000e90, reply direction”

 

Apply destination NAT to inverse source NAT action:

id=20085 trace_id=213 func=  ip_session_run_tuple line=1516 msg=”DNAT 192.168.11.59:31925-

>192.168.3.221:1487″

 

 

Flow trace output example – IPsec (policy-based)

id=20085 trace_id=1 msg=”vd-root received a packet(proto=1, 10.72.55.240:1->10.71.55.10:8)

from internal.”

id=20085 trace_id=1 msg=”allocate a new session-00001cd3″

id=20085 trace_id=1 msg=”find a route: gw-66.236.56.230 via wan1″ id=20085 trace_id=1 msg=”Allowed by Policy-2: encrypt”

id=20085 trace_id=1 msg=”enter IPsec tunnel-RemotePhase1″

id=20085 trace_id=1 msg=”encrypted, and send to 15.215.225.22 with source 66.236.56.226″ id=20085 trace_id=1 msg=”send to 66.236.56.230 via intf-wan1“

id=20085 trace_id=2 msg=”vd-root received a packet (proto=1, 10.72.55.240:1-1071.55.10:8)

from internal.”

id=20085 trace_id=2 msg=”Find an existing session, id-00001cd3, original direction” id=20085 trace_id=2 msg=”enter IPsec =”encrypted, and send to 15.215.225.22 with source

66.236.56.226“ tunnel-RemotePhase1″

id=20085 trace_id=2 msgid=20085 trace_id=2 msg=”send to 66.236.56.230 via intf-wan1″

 

Packet sniffing and packet capture

FortiOS devices can sniff packets using commands in the CLI or capture packets using the web-based manager. The differences between the two methods are not large.

Packet sniffing in the CLI is well suited for spot checking traffic from the CLI, but if you have complex filters to enter it can be a lot of work to enter them each time. You can also save the sniffing output; however, you must log to a file and then analyze the file later by hand.

Packet capture in the web-based manager makes it easy to set up multiple filters at once and just run one or two as you need them. You also have controls to start and stop capturing as you wish. Packet capture output is downloaded to your local computer as a *.pcap file which requires a third party application to read the file, such as Wireshark. This method is useful to send Fortinet support information to help resolve an issue.

 

Features                                                    Packet sniffing                         Packet capture

Command location                                       CLI                                               web-based manager

 

Third party software required                        puTTY to log plaintext output      Wireshark to read *.pcap files

 

Read output in plain text file                         yes                                               no

 

Read output as *.pcap file using Wire-

shark                                                             no                                                yes

 

Easily configure single quick and simple filter

yes                                               no

 

Record packet interface                                yes                                               no

 

 

Features                                                    Packet sniffing                         Packet capture

Configure complex sniffer filters on mul- tiple interface

no                                                yes

 

sniff IPv6                                                       hard                                             easy

 

sniff non-IP packets                                      no                                                yes

 

Filter packets by protocol and/or port            easy                                             easy

 

Filter packets by source and/or des- tination address

easy                                             easy

 

 

Packet sniffing

Before you start sniffing packets on the CLI, you should be prepared to capture the output to a file — there can be huge amounts of data that you will not be able to see without saving it to a file. One method is to use a terminal program like puTTY to connect to the FortiGate unit’s CLI. Then once the packet sniffing count is reached you can end the session and analyze the output in the file.

Details within packets passing through particular interfaces can be displayed using the packet sniffer with the following command:

diag sniffer packet <interface> <filter> <verbose> <count> <tsformat>

The <interface> value is required, with the rest being optional. If not included the default values will be “none”.

For example the simplest valid sniffer command would be:

diag sniffer packet any

 

The <interface> value can be any physical or virtual interface name. Use any to sniff packets on all interfaces.

 

The <filter> value limits the display of packets using filters, including Berkeley Packet Filtering (BPF) syntax. The <filter> value must be enclosed in quotes.

‘[[src|dst] host <host_name_or_IP1>] [[src|dst] host <host_name_or_IP2>] [[arp|ip|ip6|gre|esp|udp|tcp] [port_no]] [[arp|ip|ip6|gre|esp|udp|tcp] [port_no]]‘

 

If a second host is specified in the filter, only the traffic between the two hosts will be displayed. Optionally, you can use logical OR to match only one of the hosts, or match one of multiple protocols or ports. When defining a port, there are up to two parts — protocol and port number.

For example, to display UDP 1812 traffic or TCP 8080 traffic, use the following:

‘udp port 1812 or tcp port 8080’

 

To display all IP traffic that has a source of 192.168.1.2 and a destination of 192.168.2.3:

‘ip src host 192.168.1.2 and dst host 192.168.2.3’

 

The <verbose> option allows different levels of information to be displayed. The verbose levels include:

1 Print header of packets

2 Print header and data from the IP header of the packets

3 Print header and data from the Ethernet header of the packets

4 Print header of packets with interface name

5 Print header and data from ip of packets with interface name

6 Print header and data from ethernet of packets with interface name

 

The <count> value indicates the number of packets to sniff before stopping. If this variable is not included, or is set to zero, the sniffer will run until you manually halt it with Ctrl-C.

 

The <tsformat> value define the format of timestamp. It can be:

 

a: absolute UTC time, yyyy-mm-dd hh:mm:ss.ms

 

l: absolute LOCAL time, yyyy-mm-dd hh:mm:ss.ms otherwise: relative to the start of sniffing, ss.ms

 

Packet capture

FortiOS 5.2 includes packet capture to the web-based manager. The FortiGate unit must have a disk and then capture-packet feature can be enabled from the CLI within the firewall policy as below:

config firewall policy edit <id>

set capture-packet enable end

To configure packet capture filters, go to System > Network > Packet Capture. When you add a packet capture filter, enter the following information and select OK.

 

Interface                                     Select the interface to sniff from the dropdown menu.

You must select one interface. You cannot change the interface without deleting the filter and creating a new one, unlike the other fields.

 

Max Packets to Capture

Enter the number of packets to capture before the filter stops.

 

This number cannot be zero. You can halt the capturing before this number is reached.

 

Enable Filters                            Select this option to specify your filter fields

 

Host(s)

Enter one or more hosts IP address

 

Separate multiple hosts with commas. Enter a range using a dash without spaces, for example 172.16.1.5-172.16.1.15 or enter a subnet.

 

 

Port(s)                                         Enter one or more ports to capture on the selected interface.

 

Separate multiple ports with commas. Enter a range using a dash without spaces, for example 88-90

 

 

VLAN(s)

Enter one or more vlans (if there is any).

Separate multiple vlans with commas.

 

Protocol                                      Enter one or more protocol. Separate multiple protocol with commas. Enter a range using a dash without spaces, for example 1-6, 17, 21-25

 

Include IPv6 packets                Select this option if you are troubleshooting IPv6 networking, or if your net- work uses IPv6. Otherwise, leave it disabled.

 

Capture Non-IP packets           The protocols available in the list are all IP based except for ICMP (ping).

To capture non-IP based packets select this feature. Some examples of non-IP packets include IPsec, IGMP, ARP, and as mentioned ICMP.

 

If you select a filter and go back to edit it, you have the added option of starting and stopping packet capture in the edit window, or downloading the captured packets. You can also see the filter status and the number of packets captured.

You can also select the filter and select Start to start capturing packets. While the filter is running, you will see the number of captured packets increasing until it reaches the max packet count or you select Stop. While the filter is running you cannot download the output file.

When the packet capture is complete, you can select Download to send the packet capture filter captured packets to your local computer as a *.pcap file. To read this file format, you will need to use Wireshark or a similar third party application. Using this tool you will have extensive analytics available to you and the full contents of the packets that were captured.

 

NPU based interfaces

Many Fortinet products contain network processors such as NP1, NP2, and NP4 network processors. Therefore offloading requirements, vary by network processor model.

When using the NPU-based interfaces, only the initial session setup will be seen through the diag debug flow command. If the session is correctly programmed into the ASIC (fastpath), the debug flow command will no longer see the packets arriving at the CPU. If the NPU functionality is disabled, the CPU will see all the packets, however, this should only be used for troubleshooting purposes.

First, obtain the NP4 id and the port numbers with the following command:

diag npu np4 list

 

Sample output:

ID Model Slot Interface

0 On-board port1 fabric1 fabric3 fabric5

1 On-board fabric2 port2 base2 fabric4

 

Run the following commands:

diag npu np4 fastpaf th disable <dev_id>

 

(where dev_id is the NP4 number) Then, run this command:

diag npu np4 fastpath-sniffer enable port1

 

Sample output:

NP4 Fast Path Sniffer on port1 enabled

This will cause all traffic on port1 of NP4 to be sent to the CPU meaning a standard sniffer trace can be taken and other diag commands should work if it was a standard CPU driven port.

These commands are only for the newer NP4 interfaces.

 

Debug command

Debug output provides continuous, real-time event information. Debugging output continues until it is explicitly stopped or until the unit is rebooted. Debugging output can affect system performance and will be continually generated even though output might not be displayed in the CLI console.

Debug information displayed in the console will scroll in the console display and may prevent CLI commands from being entered, for example, the command to disable the debug display. To turn off debugging output as the display is scrolling by, press the á key to recall the recent diag debug command, press backspace, and type “0”, followed by Enter.

Debug output display is enabled using the following command:

diag debug enable

 

When finished examining the debug output, disable it using:

diag debug disable

 

Once enabled, indicate the debug information that is required using this command:

diag debug <option> <level>

 

Debug command options include the following:

enable                  Enable debug output.

disable                 Disable debug output.

info                    Show active debug level settings.

reset                   Reset all debug level to default.

report                  Report for tech support.

crashlog               Crash log info.

config-error-log       Configure error log info.

sql-log-error          SQL log database error info.

application            application.

kernel                  kernel.

remote-extender        remote-extender.

console                 console.

cli                     Debug CLI.

cmdb-trace             Trace CLI.

rating                  Display rating info.

authd                   Authentication daemon.

fsso-polling           FSSO active directory poll module.

flow                    Trace packet flow in kernel.

urlfilter              urlfilter.

admin                   Admin user.

 

The debug level can be set at the end of the command. Typical values are 2 and 3, for example:

diag debug application DHCPS 2

diag debug application spamfilter 2

 

Fortinet support will advise which debugging level to use.

Timestamps can be enabled to the debug output using the following command:

diag debug console timestamp enable

 

Debug output example

This example shows the IKE negotiation for a secure logging connection from a FortiGate unit to a FortiAnalyzer system.

diag debug reset

diag vpn ike log-filter src-addr4 192.168.11.2 diag debug enable

 

Sample Output:

FGh_FtiLog1: IPsec SA connect 0 192.168.11.2->192.168.10.201:500, natt_mode=0 rekey=0 phase2=FGh_FtiLog1

FGh_FtiLog1: using existing connection, dpd_fail=0

FGh_FtiLog1: found phase2 FGh_FtiLog1

FGh_FtiLog1: IPsec SA connect 0 192.168.11.2 -> 192.168.10.201:500 negotiating

FGh_FtiLog1: overriding selector 225.30.5.8 with 192.168.11.2

FGh_FtiLog1: initiator quick-mode set pfs=1536…

FGh_FtiLog1: try to negotiate with 1800 life seconds.

FGh_FtiLog1: initiate an SA with selectors: 192.168.11.2/0.0.0.0->192.168.10.201, ports=0/0, protocol=0/0

Send IKE Packet(quick_outI1):192.168.11.2:500(if0) -> 192.168.10.201:500, len=348

Initiator: sent 192.168.10.201 quick mode message #1 (OK) FGh_FtiLog1: set retransmit: st=168, timeout=6.

 

In this example:

192.168.11.2->192.168.10.201:500      Source and Destination gateway IP address

dpd_fail=0                              Found existing Phase 1

pfs=1536…                            Create new Phase 2 tunnel

 

The execute tac report command

exec tac report is an execute command that runs an exhaustive series of diagnostic commands. It runs commands that are only needed if you are using certain features like HA, VPN tunnels, or a modem. The report takes a few minutes to complete due to the amount of output generated. If you have your CLI output logged to a file, you can run this command to familiarize yourself with the CLI commands involved.

When you call Fortinet Customer Support, you will be asked to provide information about your unit and its current state using the output from this CLI command.

Other commands

 

ARP table

To view the ARP cache, use the following command:

get sys arp

To view the ARP cache in the system, use this command:

diag ip arp list

 

Sample output:

index=14 ifname=internal 224.0.0.5 01:00:5e:00:00:05 state=00000040 use=72203 confirm=78203 update=72203 ref=1

index=13 ifname=dmz 192.168.3.100 state=00000020 use=1843 confirm=650179 update=644179 ref=2 ? VIP

index=13 ifname=dmz 192.168.3.109 02:09:0f:78:69:ff state=00000004 use=71743 confirm=75743 update=75743 ref=1

index=14 ifname=internal 192.168.11.56 00:1c:23:10:f8:20 state=00000004 use=10532 confirm=10532 update=12658 ref=4

 

To remove the ARP cache, use this command:

execute clear system arp table

 

To remove a single ARP entry, use:

diag ip arp delete <interface name> <IP address>

 

To add static ARP entries, use the following command:

config system arp-table

 

Time and date settings

Check time and date settings for log message timestamp synchronization (the Fortinet support group may request this) and for certificates that have a time requirement to check for validity. Use the following commands:

execute time

current time is: 12:40:48

last ntp sync:Thu Mar 16 12:00:21 2006 execute date

current date is: 2006-03-16

 

To force synchronization with an NTP server, toggle the following command:

Config system ntp

set ntpsync enable/disable end

 

If all devices have the same time, it helps to correlate log entries from different devices.

 

IP address

There may be times when you want to verify the IP addresses assigned to the FortiGate unit interfaces are what you expect them to be. This is easily accomplished from the CLI using the following command.

diag ip address list

 

The output from this command lists the IP address and mask if available, the index of the interface (a sort of ID number) and the devname is the name of the interface. While physical interface names are set, virtual interface names can vary. Listing all the virtual interface names is a good use of this command. For vsys_ha and vsys_ fgfm, the IP addresses are the local host — these are internally used virtual interfaces.

# diag ip address list

IP=10.31.101.100->10.31.101.100/255.255.255.0 index=3 devname=internal

IP=172.20.120.122->172.20.120.122/255.255.255.0 index=5 devname=wan1

IP=127.0.0.1->127.0.0.1/255.0.0.0 index=8 devname=root IP=127.0.0.1->127.0.0.1/255.0.0.0 index=11 devname=vsys_ha IP=127.0.0.1->127.0.0.1/255.0.0.0 index=13 devname=vsys_fgfm


Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!

Chapter 26 – Troubleshooting

Chapter 26 – Troubleshooting

 

This handbook chapter describes concepts of troubleshooting and solving issues that may occur with FortiGate units.

This FortiOS Handbook chapter contains the following chapters:

Verifying FortiGate admin access security explains how to verify and configure administrative access. Troubleshooting resources walks you through Fortinet’s resources for troubleshooting.

Troubleshooting tools describes some of the basic commands and parts of FortiOS that can help you with troubleshooting.

Troubleshooting methodologies walks you through best practice concepts of FortiOS troubleshooting.

Technical Support Organization Overview describes how Fortinet Support operates, what they will need from you if you contact them, and what you can expect in general.

Common questions answers most of the common questions.

 

Troubleshooting methodologies

Before you begin troubleshooting anything but the most minor issues, you need to prepare. Doing so will shorten the time to solve your issue. This section helps to explain how you prepare before troubleshooting, as well as creating a troubleshooting plan and contacting support.

 

This section contains the following topics:

  • Establish a baseline
  • Define the problem
  • Gathering Facts
  • Create a troubleshooting plan
  • Obtain any required additional equipment
  • Ensure you have administrator level access to required equipment
  • Contact Fortinet customer support for assistance

 

Establish a baseline

FortiGate units operate at all layers of the OSI model. For this reason troubleshooting problems can become complex. If you establish a normal operation parameters, or baseline, for your system before the problem occurs it will help reduce the complexity when you are troubleshooting.

Many of the guiding questions in the following sections are some form of comparing the current problem situation to normal operation on your FortiGate unit. For this reason it is a best practice that you know what your normal operating status is, and have a record of it you can refer to. This can easily be accomplished by monitoring the system performance with logs, SNMP tools, or regularly running information gathering commands and saving the output. This regular operation data will show trends, and enable you to see when changes happen and there may be a problem.

Back up your FortiOS configuration on a regular basis. This is a good practice for every- day as well as when troubleshooting. You can restore the backed up configuration when needed and save the time and effort of re-creating it from the factory default set- tings.

Some fundamental CLI commands you can use to obtain normal operating data for your system:

get system status              Displays versions of firmware and FortiGuard engines, and other system information.

 

get system performance status

Displays CPU and memory states, average network usage, aver- age sessions and session setup rate, virus caught, IPS attacks blocked, and uptime.

 

get hardware memory            Displays informations about memory

 

get system session status      Displays total number of sessions

 

get router info routing-table all

Displays all the routes in the routing table including their type, source, and other useful data.

 

get ips session                Displays memory used and max available to IPS as well and counts.

 

get webfilter ftgd-statistics Displays list of FortiGuard related counts of status, errors, and other data.

 

diagnose firewall statistic show

Displays the amount of network traffic broken down into cat- egories such as email, VoIP, TCP, UDP, IM, Gaming, P2P, and Streaming.

 

diag system session list       Displays current detailed sessions list

 

show system dns                Displays configured DNS servers

 

diag sys ntp status            Displays informations about ntp servers

 

These commands are just a sample. Feel free to include any extra information gathering commands that apply to your system. For example if you have active VPN connections, record information about them using the get vpn * series of commands.

For an extensive snapshot of your system, run the CLI command used by TAC to gather extensive information about a system — exec tac report. It runs many diagnostic commands that are for specific configurations. This means no matter what features you are using, this command will record their current state. Then if you need to perform troubleshooting at a later date, you can run the same command again and compare the differences to quickly locate suspicious output you can investigate.

 

Define the problem

The following questions can help determine the scope of the problem and isolate it:

 

  • What is the problem?

Do not assume that the problem is being experienced is the actual problem. First determine that the problem does not lie elsewhere before starting to troubleshoot the FortiGate device.

  • Has it ever worked before?

If the device never worked from the first day, you may not want to spend time troubleshooting something that could well be defective. See “Troubleshooting bootup”.

  • Can the problem be reproduced at will or is it intermittent?

If the problem is intermittent, it may be dependent on system load. Also an intermittent problem can be very difficult to troubleshoot due to the difficulty reproducing the issue.

  • What has changed?

Do not assume that nothing has changed in the network. Use the FortiGate event log to see if any configuration changes were made. The change could be in the operating environment, for example, a gradual increase in load as more sites are forwarded through the firewall.

If something has changed, see what the affect is if the change is rolled back.

  • Determine the scope of the problem – after you have isolated the problem what applications, users, devices, and operating systems does it effect?

 

Before you can solve a problem, you need to understand it. Often this step can be the longest in this process. Ask questions such as:

  • What is not working? Be specific.
  • Is there more than one thing not working?
  • Is it partly working? If so, what parts are working?
  • Is it a connectivity issue for the whole device, or is there an application that isn’t reaching the Internet? Be as specific as possible with your answers, even if it takes awhile to find the answers.

These questions will help you define the problem. Once the problem is defined, you can search for a solution and then create a plan on how to solve it.

 

Gathering Facts

Fact gathering is an important part of defining the problem. Record the following information as it applies to the problem:

  • Where did the problem occur?
  • When did the problem occur and to whom?
  • What components are involved?
  • What is the affected application?
  • Can the problem be traced using a packet sniffer?
  • Can the problem be traced in the session table or using system debugging?
  • Can log files be obtained that indicate a failure has occurred?

Answers to these questions will help you narrow down the problem, and what you have to check during your troubleshooting. The more things you can eliminate, the fewer things you need to check during troubleshooting. For this reason, be as specific and accurate as you can while gathering facts.

 

Create a troubleshooting plan

Once you have defined the problem, and searched for a solution you can create a plan to solve that problem. Even if your search didn’t find a solution to your problem you may have found some additional things to check to further define your problem.

The plan should list all the possible causes of the problem that you can think of, and how to test for each possible cause.

Your troubleshooting plan will act as a checklist so that you know what you have tried and what is left to check. This is important to have if more than one person will be doing the troubleshooting. Without a written plan, people will become easily confused and steps will be skipped. Also if you have to hand over the problem to someone else, providing them with a detailed list of what data has been gathered and what solutions have been already tried demonstrates a good level of professionalism.

Be ready to add to your plan as needed. After you are part way through, you may discover that you forgot some tests or a test you performed discovered new information. This is normal.

Also if you contact support, they will require information about your problem as well as what you have already tried to fix the problem. This should all be part of your plan.

 

Providing Supporting Elements

If the Fortinet Technology Assistance Center (TAC) needs to be contacted to help you with your issue, be prepared to provide the following information:

  • The firmware build version (use the get system status command)
  • A network topology diagram
  • A recent configuration file
  • Optionally, a recent debug log
  • Tell the support team what troubleshooting steps have already been performed and the results.

Do not provide the output from exec tac report unless Support requests it. The output from that command is very large and is not required in many cases.

For additional information about contacting Fortinet Customer Support, see Technical Support Organization Overview.

All of this is your troubleshooting plan.

 

Obtain any required additional equipment

You may require additional networking equipment, computers, or other equipment to test your solution. Normally network administrators have additional networking equipment available either to loan you, or a lab where you can bring the FortiGate unit to test.

If you do not have access to equipment, check for shareware applications that can perform the same task. Often there are software solutions when hardware is too expensive.

 

Ensure you have administrator level access to required equipment

Before troubleshooting your FortiGate unit, you will need administrator access to the equipment. If you are a client on a FortiGate unit with virtual domains enabled, often you can troubleshoot within your own VDOM. However, you should inform your FortiGate unit’s super admin that you will be doing troubleshooting.

Also, you may need access to other networking equipment such as switches, routers, and servers to help you test. If you do not normally have access to this equipment, contact your network administrator for assistance.

 

Contact Fortinet customer support for assistance

You have defined your problem, researched a solution, put together a plan to find the solution, and executed that plan. At this point if the problem has not been solved, its time to contact Fortinet Customer Support for assistance.

For more information, see Technical Support Organization Overview.


Having trouble configuring your Fortinet hardware or have some questions you need answered? Check Out The Fortinet Guru Youtube Channel! Want someone else to deal with it for you? Get some consulting from Fortinet GURU!