IPsec VPN troubleshooting

Posted on by Keith Leroux

Share this post:

This section contains tips to help you with some common challenges of IPsec VPNs.
A VPN connection has multiple stages that can be confirmed to ensure the connection is working
properly. It is easiest to see if the final stage is successful first since if it is successful the other stages
will be working properly. Otherwise, you will need to work back through the stages to see where the
problem is located.
When a VPN connection is properly established, traffic will flow from one end to the other as if both
ends were physically in the same place. If you can determine the connection is working properly then
any problems are likely problems with your applications.
On some FortiGate units, such as the FortiGate 94D, you cannot ping over the IPsec tunnel without
first setting a source-IP. In this scenario, you must assign an IP address to the virtual IPSEC VPN
interface. Anything sourced from the FortiGate going over the VPN will use this IP address.
If the egress/outgoing interface (determined by kernel route) has an IP address, then use the IP address
of the egress/outgoing interface. Otherwise, use the IP address of the first interface from the interface
list (that has an IP address).
The first diagnostic command worth running, in any IPsec VPN troubleshooting situation, is the
following:
diagnose vpn tunnel list

This command is very useful for gathering statistical data such as the number of packets encrypted
versus decrypted, the number of bytes sent versus received, the SPI identifier, etc. This kind of
information in the resulting output can make all the difference in determining the issue with the VPN.
Another appropriate diagnostic command worth trying is:
diagnose debug flow

This command will inform you of any lack of firewall policy, lack of forwarding route, and of policy
ordering issues.
The following is a list of such potential issues. Bear in mind that the troubleshooting suggestions below
are not exhaustive, and may not reflect your network topology.

Watch the video

The options to configure policy-based IPsec VPN are unavailable.
Go to System > Feature Select. Select Show More and turn on Policy-based IPsec VPN.
If your VPN fails to connect, check the following:
• Ensure that the pre-shared keys match exactly (see The pre-shared key does not match
(PSK mismatch error) below).
• Ensure that both ends use the same P1 and P2 proposal settings (see The SA proposals do not
match (SA proposal mismatch) below).
• Ensure that you have allowed inbound and outbound traffic for all necessary network services,
especially if services such as DNS or DHCP are having problems.
• Check that a static route has been configured properly to allow routing of VPN traffic.
• Ensure that your FortiGate unit is in NAT/Route mode, rather than Transparent.
• Check your NAT settings, enabling NAT traversal in the Phase 1 configuration while disabling
NAT in the security policy. You might need to pin the PAT/NAT session table, or use some of
kind of NAT-T keepalive to avoid the expiration of your PAT/NAT translation.
• Ensure that both ends of the VPN tunnel are using Main mode, unless multiple dial-up tunnels
are being used.
• If you have multiple dial-up IPsec VPNs, ensure that the Peer ID is configured properly on the
• FortiGate and that clients have specified the correct Local ID.
• If you are using FortiClient, ensure that your version is compatible with the FortiGate firmware
by reading the FortiOS Release Notes.
• If you are using Perfect Forward Secrecy (PFS), ensure that it is used on both peers. You can
use the diagnose vpn tunnel list command to troubleshoot this.
• Ensure that the Quick Mode selectors are correctly configured. If part of the setup currently
uses firewall addresses or address groups, try changing it to either specify the IP addresses or
use an expanded address range. This is especially useful if the remote endpoint is not a
FortiGate device.
• If XAUTH is enabled, ensure that the settings are the same for both ends, and that the FortiGate
unit is set to Enable as Server.
• Check IPsec VPN Maximum Transmission Unit (MTU) size. A 1500 byte MTU is going to
exceed the overhead of the ESP-header, including the additional ip_header,etc. You can use the
diagnose vpn tunnel list command to troubleshoot this.
• If your FortiGate unit is behind a NAT device, such as a router, configure port forwarding for
UDP ports 500 and 4500.
• Remove any Phase 1 or Phase 2 configurations that are not in use. If a duplicate instance of the
VPN tunnel appears on the IPsec Monitor, reboot your FortiGate unit to try and clear the entry.
If you are still unable to connect to the VPN tunnel, run the following diagnostic command in the CLI:
diagnose debug application ike -1
diagnose debug enable

The resulting output may indicate where the problem is occurring. When you are finished, disable the
diagnostics by using the following command:
diagnose debug reset
diagnose debug disable

The VPN tunnel goes down frequently.

If your VPN tunnel goes down often, check the Phase 2 settings and either increase the Keylife value
or enable Autokey Keep Alive.

The pre-shared key does not match (PSK mismatch error).

It is possible to identify a PSK mismatch using the following combination of CLI commands:
diag vpn ike log filter name <phase1-name>
diag debug app ike -1
diag debug enable

This will provide you with clues as to any PSK or other proposal issues. If it is a PSK mismatch, you
should see something similar to the following output:
ike 0:TRX:322: PSK auth failed: probable pre-shared key mismatch
ike Negotiate SA Error:

The SA proposals do not match (SA proposal mismatch).

The most common problem with IPsec VPN tunnels is a mismatch between the proposals offered
between each party. Without a match and proposal agreement, Phase 1 can never establish. Use the
following command to show the proposals presented by both parties.
diag debug app ike -1
diag debug enable

The resulting output should include something similar to the following, where blue represents the
remote VPN device, and green represents the local FortiGate.
responder received SA_INIT msg
incoming proposal:
proposal id = 1:
protocol = IKEv2:
encapsulation = IKEv2/none
type=ENCR, val=AES_CBC (key_len = 256)
type=INTEGR, val=AUTH_HMAC_SHA_96
type=PRF, val=PRF_HMAC_SHA
type=DH_GROUP, val=1536.
proposal id = 2:
protocol = IKEv2:
encapsulation = IKEv2/none
type=ENCR, val=3DES_CBC
 type=INTEGR, val=AUTH_HMAC_SHA_2_256_128
type=PRF, val=PRF_HMAC_SHA2_256
type=DH_GROUP, val=1536.
proposal id = 1:
protocol = IKEv2:
encapsulation = IKEv2/none
type=ENCR, val=AES_CBC (key_len = 128)
type=INTEGR, val=AUTH_HMAC_SHA_96
type=PRF, val=PRF_HMAC_SHA
type=DH_GROUP, val=1536.

Pre-existing IPsec VPN tunnels need to be cleared.

Should you need to clear an IKE gateway, use the following commands:
diagnose vpn ike restart
diagnose vpn ike gateway clear

LAN interface connection

To confirm whether a VPN connection over LAN interfaces has been configured correctly, issue a ping
or traceroute command on the network behind the FortiGate unit to test the connection to a computer
on the remote network. If the connection is properly configured, a VPN tunnel will be established
automatically when the first data packet destined for the remote network is intercepted by the FortiGate
unit.
If the ping or traceroute fail, it indicates a connection problem between the two ends of the tunnel. This
may or may not indicate problems with the VPN tunnel. You can confirm this by going to Monitor
> IPsec Monitor where you will be able to see your connection. A green arrow means the tunnel is up
and currently processing traffic. A red arrow means the tunnel is not processing traffic, and this VPN
connection has a problem.

Dialup connection
A dialup VPN connection has additional steps. To confirm that a VPN between a local network and a
dialup client has been configured correctly, at the dialup client, issue a ping command to test the
connection to the local network. The VPN tunnel initializes when the dialup client attempts to connect.
If the ping or traceroute fail, it indicates a connection problem between the two ends of the tunnel. This
may or may not indicate problems with the VPN tunnel, or dialup client. As with the LAN connection,
confirm the VPN tunnel is established by checking Monitor > IPsec Monitor.

Troubleshooting VPN connections

If you have determined that your VPN connection is not working properly through troubleshooting, the
next step is to verify that you have a Phase2 connection.
If traffic is not passing through the FortiGate unit as you expect, ensure the traffic does not contain
IPcomp packets (IP protocol 108, RFC 3173). FortiGate units do not allow IPcomp packets, they
compress packet payload, preventing it from being scanned.
Testing Phase 1 and 2 connections is a bit more difficult than testing the working VPN. This is because
they require diagnose CLI commands. These commands are typically used by Fortinet customer
support to discover more information about your FortiGate unit and its current configuration.
Before you begin troubleshooting, you must:
• Configure FortiGate units on both ends for interface VPN
• Record the information in your VPN Phase 1 and Phase 2 configurations – for our example here
the remote IP address is 10.11.101.10 and the names of the phases are Phase 1 and Phase 2
• Install a telnet or SSH client such as putty that allows logging of output
• Ensure that the admin interface supports your chosen connection protocol so you can connect to
your FortiGate unit admin interface.
For this example, default values were used unless stated otherwise.

To get diagnose information for the VPN connection – CLI

1. Log into the CLI as admin with the output being logged to a file.
2. Stop any diagnose debug sessions that are currently running with the CLI command
diagnose debug disable

3. Clear any existing log-filters by running

diagnose vpn ike log-filter clear

4. Set the log-filter to the IP address of the remote computer (10.11.101.10). This filters out all
VPN connections except ones to the IP address we are concerned with. The command is
diagnose vpn ike log-filter dst-addr4 10.11.101.10.

5. Set up the commands to output the VPN handshaking. The commands are:
diagnose debug app ike 255
diagnose debug enable

6. Have the remote FortiGate initiate the VPN connection in the web-based manager by going
to VPN > IPsec Tunnels and selecting Bring up.
This makes the remote FortiGate the initiator and the local FortiGate becomes the responder.
Establishing the connection in this manner means the local FortiGate will have its configuration
information as well as the information the remote computer sends. Having both sets of
information locally makes it easier to troubleshoot your VPN connection.
7. Watch the screen for output, and after roughly 15 seconds enter the following CLI command to
stop the output.
diagnose debug disable
 8. If needed, save the log file of this output to a file on your local computer. Saving the output to a
file can make it easier to search for a particular phrase, and is useful for comparisons.

To troubleshoot a phase1 VPN connection

Using the output from To get diagnose information for the VPN connection – CLI, search for the word
proposal in the output. It may occur once indicating a successful connection, or it will occur two or
more times for an unsuccessful connection — there will be one proposal listed for each end of the
tunnel and each possible combination in their settings. For example if 10.11.101.10 selected both
Diffie-Hellman Groups 1 and 5, that would be at least 2 proposals set.
A successful negotiation proposal will look similar to:
IPsec SA connect 26 10.12.101.10->10.11.101.10:500
config found
created connection: 0x2f55860 26 10.12.101.10->10.11.101.10:500
IPsec SA connect 26 10.12.101.10->10.11.101.10:500 negotiating
no suitable ISAKMP SA, queuing quick-mode request and initiating ISAKMP SA
negotiation
initiator: main mode is sending 1st message...
cookie 3db6afe559e3df0f/0000000000000000
out [encryption]
sent IKE msg (ident-i1send): 10.12.101.10:500->10.11.101.10:500, len=264,
id=3db6afe559e3df0f/0000000000000000
diaike 0: comes 10.12.101.1:500->10.11.101.1:500,ifindex=26....

Note the phrase “initiator: main mode is sending 1st message...” which shows
you the handshake between the ends of the tunnel is in progress. Initiator shows the remote unit is
sending the first message.

VPN troubleshooting tips

Attempting hardware offloading beyond SHA1
If you are trying to off-load VPN processing to a network processing unit (NPU), remember that only
SHA1 authentication is supported. For high levels of authentication such as SHA256, SHA384, and
SHA512 hardware offloading is not an option — all VPN processing must be done in software.

Enable/disable IPsec ASIC-offloading

Much like NPU-offload in IKE phase1 configuration, you can enable or disable the usage of ASIC
hardware for IPsec Diffie-Hellman key exchange and IPsec ESP traffic. By default hardware offloading
is used. For debugging purposes, sometimes it is best for all the traffic to be processed by software.
config sys global
set ipsec-asic-offload [enable|disable]
end

Check Phase 1 proposal settings

Ensure that both sides have at least one Phase 1 proposal in common. Otherwise they will not connect.
If there are many proposals in the list, this will slow down the negotiating of Phase 1. If its too slow,
the connection may timeout before completing. If this happens, try removing some of the unused
proposals.
NPU offloading is supported when the local gateway is a loopback interface.

Check your routing

If routing is not properly configured with an entry for the remote end of the VPN tunnel, traffic will not
flow properly. You may need static routes on both ends of the tunnel. If routing is the problem, the
proposal will likely setup properly but no traffic will flow.

Try enabling XAuth

If one end of an attempted VPN tunnel is using XAuth and the other end is not, the connection attempt
will fail. The log messages for the attempted connection will not mention XAuth is the reason, but
when connections are failing it is a good idea to ensure both ends have the same XAuth settings. If you
do not know the other end’s settings enable or disable XAuth on your end to see if that is the problem.

General troubleshooting tips

Most connection failures are due to a configuration mismatch between the FortiGate unit and the
remote peer. In general, begin troubleshooting an IPsec VPN connection failure as follows:
1. Ping the remote network or client to verify whether the connection is up.
2. Traceroute the remote network or client. If DNS is working, you can use domain names.
Otherwise use IP addresses.
3. Check the routing behind the dialup client. Routing problems may be affecting DHCP. If this
appears to be the case, configure a DHCP relay service to enable DHCP requests to be relayed
to a DHCP server on or behind the FortiGate server.
4. Verify the configuration of the FortiGate unit and the remote peer. Check the following IPsec
parameters:
• The mode setting for ID protection (main or aggressive) on both VPN peers must be identical.
• The authentication method (preshared keys or certificates) used by the client must be supported
on the FortiGate unit and configured properly.
• If preshared keys are being used for authentication purposes, both VPN peers must have
identical preshared keys.
• The remote client must have at least one set of Phase 1 encryption, authentication, and Diffie-
Hellman settings that match corresponding settings on the FortiGate unit.
• Both VPN peers must have the same NAT traversal setting (enabled or disabled).
• The remote client must have at least one set of Phase 2 encryption and authentication algorithm
settings that match the corresponding settings on the FortiGate unit.
• If you are using manual keys to establish a tunnel, the Remote SPI setting on the FortiGate unit
must be identical to the Local SPI setting on the remote peer, and vise versa.
 5. To correct the problem, see the following table.

VPN troubleshooting tips

Configuration problem
Correction
Mode settings do not match. Select complementary mode settings.
Check Phase 1 configuration. Depending on the Remote
Gateway and Authentication Method settings, you have a choice
of options to authenticate FortiGate dialup clients or VPN peers
Peer ID or certificate name of the by ID or certificate name.
remote peer or dialup client is not
recognized by FortiGate VPN server. If you are configuring authentication parameters for FortiClient
dialup clients, refer to the Authenticating FortiClient Dialup
Clients Technical Note.

Preshared keys do not match. Reenter the preshared key.

Phase 1 or Phase 2 key exchange Make sure that both VPN peers have at least one set of
proposals are mismatched. proposals in common for each phase.
NAT traversal settings are
Select or clear both options as required.
mismatched.

A word about NAT devices

When a device with NAT capabilities is located between two VPN peers or a VPN peer and a dialup
client, that device must be NAT traversal (NAT-T) compatible for encrypted traffic to pass through the
NAT device.