IPsec VPN in the web-based manager

IPsec VPN in the web-based manager

To configure an IPsec VPN, use the general procedure below. With these steps, your FortiGate unit will automatically generate unique IPsec encryption and authentication keys. If a remote VPN peer or client requires a specific IPsec encryption or authentication key, you must configure your FortiGate unit to use manual keys instead.

1. Define Phase 1 parameters to authenticate remote peers and clients for a secure connection. See IPsec VPN in the web-based manager on page 1611.

2. Define Phase 2 parameters to create a VPN tunnel with a remote peer or dialup client. See IPsec VPN in the web- based manager on page 1611.

3. Create a security policy to permit communication between your private network and the VPN. Policy-based VPNs have an action of IPSEC, where for interface-based VPNs the security policy action is ACCEPT. See Defining VPN security policies on page 1648.

The FortiGate unit implements the Encapsulated Security Payload (ESP) protocol. Internet Key Exchange (IKE) is performed automatically based on pre-shared keys or X.509 digital certificates. Interface mode, supported in NAT mode only, creates a virtual interface for the local end of a VPN tunnel.

This chapter contains the following sections: Phase 1 configuration

Phase 2 configuration


IPsec Monitor


Phase 1 configuration

To begin defining the Phase 1 configuration, go to VPN > IPsec Tunnels and select Create New. Enter a unique descriptive name for the VPN tunnel and follow the instructions in the VPN Creation Wizard.

The Phase 1 configuration mainly defines the ends of the IPsec tunnel. The remote end is the remote gateway with which the FortiGate unit exchanges IPsec packets. The local end is the FortiGate interface that sends and receives IPsec packets.

If you want to control how the IKE negotiation is processed when there is no traffic, as well as the length of time the FortiGate unit waits for negotiations to occur, you can use the negotiation-timeout and auto- negotiate commands in the CLI.

For more information, refer to Phase 2 parameters on page 1642 and Phase 2 parameters on page 1642.

Name                                           Type a name for the Phase 1 definition. The maximum name length is 15 characters for an interface mode VPN, 35 characters for a policy-based VPN. If Remote Gateway is Dialup User, the maximum name length is further reduced depending on the number of dialup tunnels that can be established: by 2 for up to 9 tunnels, by 3 for up to 99 tunnels, 4 for up to 999 tunnels, and so on.

For a tunnel mode VPN, the name normally reflects where the remote con- nection originates. For a route-based tunnel, the FortiGate unit also uses the name for the virtual IPsec interface that it creates automatically.

Select the category of the remote connection:

Remote Gateway

Static IP Address — If the remote peer has a static IP address.

Dialup User — If one or more FortiClient or FortiGate dialup clients with dynamic IP addresses will connect to the FortiGate unit.

Dynamic DNS — If a remote peer that has a domain name and sub-scribes to a dynamic DNS service will connect to the FortiGate unit.

IP Address                                 If you selected Static IP Address, enter the IP address of the remote peer.

Dynamic DNS                            If you selected Dynamic DNS, enter the domain name of the remote peer.

Local Interface                          This option is available in NAT mode only. Select the name of the interface through which remote peers or dialup clients connect to the FortiGate unit.

By default, the local VPN gateway IP address is the IP address of the inter- face that you selected.

Main mode — the Phase 1 parameters are exchanged in multiple rounds with encrypted authentication information.

Aggressive mode — the Phase 1 parameters are exchanged in single message with authentication information that is not encrypted.


When the remote VPN peer has a dynamic IP address and is authenticated by a pre-shared key, you must select Aggressive mode if there is more than one dialup phase1 configuration for the interface IP address.

When the remote VPN peer has a dynamic IP address and is authenticated by a certificate, you must select Aggressive mode if there is more than one Phase 1 configuration for the interface IP address and these Phase 1 con- figurations use different proposals.

Authentication Method            Select Preshared Key or RSA Signature.

Preshared Key

If you selected Preshared Key, enter the pre-shared key that the FortiGate unit will use to authenticate itself to the remote peer or dialup cli- ent during Phase 1 negotiations. You must define the same key at the remote peer or client. The key must contain at least 6 printable characters. For optimum protection against currently known attacks, the key must con- sist of a minimum of 16 randomly chosen alphanumeric characters.

Certificate Name                        If you selected RSA Signature, select the name of the server certificate that the FortiGate unit will use to authenticate itself to the remote peer or dialup client during Phase 1 negotiations. For information about obtaining and loading the required server certificate, see the FortiOS User Authentic- ation guide.

Peer Options

Peer options are available to authenticate VPN peers or clients, depending on the Remote Gateway and Authentication Method settings.

Any peer ID                                Accept the local ID of any remote VPN peer or client. The FortiGate unit does not check identifiers (local IDs). You can set Mode to Aggressive or Main.

You can use this option with RSA Signature authentication. But, for highest security, configure a PKI user/group for the peer and set Peer Options to Accept this peer certificate only.

This option is available when Aggressive Mode is enabled. Enter the iden- tifier that is used to authenticate the remote peer. This identifier must match the Local ID that the remote peer’s administrator has configured.


This peer ID

If the remote peer is a FortiGate unit, the identifier is specified in the LocaID field of the Advanced Phase 1 configuration.

If the remote peer is a FortiClient user, the identifier is specified in the Local ID field, accessed by selecting Config in the Policy section of the VPN connection’s Advanced Settings.

Peer ID from dialup group       Authenticate multiple FortiGate or FortiClient dialup clients that use unique identifiers and unique pre-shared keys (or unique pre-shared keys only) through the same VPN tunnel.

You must create a dialup user group for authentication purposes. Select the group from the list next to the Peer ID from dialup group option.

You must set Mode to Aggressive when the dialup clients use unique identifiers and unique pre-shared keys. If the dialup clients use unique pre- shared keys only, you can set Mode to Main if there is only one dialup Phase 1 configuration for this interface IP address.

Phase 1 advanced configuration settings

You can use the following advanced parameters to select the encryption and authentication algorithms that the FortiGate unit uses to generate keys for the IKE exchange. You can also use the following advanced parameters to ensure the smooth operation of Phase 1 negotiations.

These settings are mainly configured in the CLI, although some options are available after the tunnel is created using the VPN Creation Wizard (using the Convert to Custom Tunnel option).

VXLAN over IPsec                  Packets with VXLAN header are encapsulated within IPsec tunnel mode.

New attributes in IPsec phase1 settings have been added.

To configure VXLAN over IPsec – CLI:

config vpn ipsec phase1-interface/phase1 edit ipsec

set interface <name>

set encapsulation vxlan/gre (new)

set encapsulation-address ike/ipv4/ipv6 (New)

set encap-local-gw4 xxx.xxx.xxx.xxx (New)

set encap-remote-gw xxx.xxx.xxx.xxx (New)

next end

You can define an idle timer for IPsec tunnels. When no traffic has passed through the tunnel for the configured idle-timeout value, the IPsec tunnel will be flushed.


IPsec tunnel idle timer

To configure IPsec tunnel idle timeout – CLI:

config vpn ipsec phase1-interface edit p1

set idle-timeout [enable | disable]

set idle-timeoutinterval <integer> //IPsec tunnel idle timeout in minutes (10 – 43200).

end end


IPv6 Version                              Select if you want to use IPv6 addresses for the remote gateway and inter- face IP addresses.

Specify an IP address for the local end of the VPN tunnel. Select one of the following:

Local Gateway IP

Main Interface IP — The FortiGate unit obtains the IP address of the interface from the network interface settings.

Specify — Enter a secondary address of the interface selected in the

Phase 1 Local Interface field.

You cannot configure Interface mode in a transparent mode VDOM.

Phase 1 Proposal                      Select the encryption and authentication algorithms used to generate keys for protecting negotiations and add encryption and authentication algorithms as required.

You need to select a minimum of one and a maximum of three com- binations. The remote peer or client must be configured to use at least one of the proposals that you define.

Select one of the following symmetric-key encryption algorithms:

DES — Digital Encryption Standard, a 64-bit block algorithm that uses a 56-bit key.

3DES — Triple-DES; plain text is encrypted three times by three keys.

AES128 — A 128-bit block algorithm that uses a 128-bit key. AES192 — A 128-bit block algorithm that uses a 192-bit key. AES256 — A 128-bit block algorithm that uses a 256-bit key.

You can select either of the following message digests to check the authen- ticity of messages during an encrypted session:

MD5 — Message Digest 5.

SHA1 — Secure Hash Algorithm 1 – a 160-bit message digest.

To specify one combination only, set the Encryption and Authentication options of the second combination to NULL. To specify a third com- bination, use the Add button beside the fields for the second combination.

DiffieHellman Group                Select one or more Diffie-Hellman groups from DH groups 1, 2, 5, and 14 through 21. At least one of the Diffie-Hellman Group settings on the remote peer or client must match one the selections on the FortiGate unit. Failure to match one or more DH groups will result in failed negotiations.

KeylifEnter the time (in seconds) that must pass before the IKE encryption key expires. When the key expires, a new key is generated without interrupting service. The keylife can be from 120 to 172 800 seconds.

Local ID                                      If the FortiGate unit will act as a VPN client and you are using peer IDs for authentication purposes, enter the identifier that the FortiGate unit will sup- ply to the VPN server during the Phase 1 exchange.

If the FortiGate unit will act as a VPN client, and you are using security cer- tificates for authentication, select the distinguished name (DN) of the local server certificate that the FortiGate unit will use for authentication pur- poses.

If the FortiGate unit is a dialup client and will not be sharing a tunnel with other dialup clients (that is, the tunnel will be dedicated to this Fortinet dia- lup client), set Mode to Aggressive.

Note that this Local ID value must match the peer ID value given for the remote VPN peer’s Peer Options.

This option supports the authentication of dialup clients. It is available for IKE v1 only.

XAutDisable — Select if you do not use XAuth.

Enable as Client — If the FortiGate unit is a dialup client, enter the user name and password that the FortiGate unit will need to authenticate itself to the remote XAuth server.

Enable as Server — This is available only if Remote Gateway is set to

Dialup User. Dialup clients authenticate as members of a dialup user group. You must first create a user group for the dialup clients that need access to the network behind the FortiGate unit.

You must also configure the FortiGate unit to forward authentication requests to an external RADIUS or LDAP authentication server.

Select a Server Type setting to determine the type of encryption method to use between the FortiGate unit, the XAuth client and the external authentication server, and then select the user group from the User Group list.

Username                                   Enter the user name that is used for authentication.

Password                                   Enter the password that is used for authentication.

NAT Traversal                            Select the check box if a NAT device exists between the local FortiGate unit and the VPN peer or client. The local FortiGate unit and the VPN peer or client must have the same NAT traversal setting (both selected or both cleared) to connect reliably.

Additionally, you can force IPsec to use NAT traversal. If NAT is set to Forced, the FortiGate will use a port value of zero when constructing the NAT discovery hash for the peer. This causes the peer to think it is behind a NAT device, and it will use UDP encapsulation for IPsec, even if no NAT is present. This approach maintains interoperability with any IPsec imple- mentation that supports the NAT-T RFC.

Keepalive Frequency               If you enabled NATtraversal, enter a keepalive frequency setting.

Dead Peer Detection                 Select this check box to reestablish VPN tunnels on idle connections and clean up dead IKE peers if required. You can use this option to receive noti- fication whenever a tunnel goes up or down, or to keep the tunnel con- nection open when no traffic is being generated inside the tunnel. For example, in scenarios where a dialup client or dynamic DNS peer connects from an IP address that changes periodically, traffic may be suspended while the IP address changes.

With Dead Peer Detection selected, you can use the config vpn ipsec phase1 (tunnel mode) or config vpn ipsec phase1- interface (interface mode) CLI command to optionally specify a retry count and a retry interval.


IKE fragmentation

UDP fragmentation can cause issues in IPsec when either the ISP or perimeter firewall(s) cannot pass or fragment the oversized UDP packets that occur when using a very large public security key (PSK). The result is that IPsec tunnels do not come up. The solution is IKE fragmentation.

For most configurations, enabling IKE fragmentation allows connections to automatically establish when they otherwise might have failed due to intermediate nodes dropping IKE messages containing large certificates, which typically push the packet size over 1500 bytes.

FortiOS will fragment a packet on sending if, and only if, all the following are true:

  • Phase 1 contains “set fragmentation enable”.
  • The packet is larger than the minimum MTU (576 for IPv4, 1280 for IPv6).
  • The packet is being re-transmitted.

By default, IKE fragmentation is enabled, but upon upgrading, any existing phase1-interface may have have “set fragmentation disable” added in order to preserve the existing behaviour of not supporting fragmentation.


To enable or disable IKE fragmentation – CLI

config vpn ipsec phase1-interface edit 1

set fragmentation [enable | disable]

next end


This entry was posted in FortiOS 5.4 Handbook and tagged , on by .

About Mike

Michael Pruett, CISSP has a wide range of cyber-security and network engineering expertise. The plethora of vendors that resell hardware but have zero engineering knowledge resulting in the wrong hardware or configuration being deployed is a major pet peeve of Michael's. This site was started in an effort to spread information while providing the option of quality consulting services at a much lower price than Fortinet Professional Services. Owns PacketLlama.Com (Fortinet Hardware Sales) and Office Of The CISO, LLC (Cybersecurity consulting firm).

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.