Description
This article provides information to help debug an encryption key mismatch.
Scope
IPSec VPN
Solution
The ESP packet invalid error is due to an encryption key mismatch after a VPN tunnel has been established. When an IPSec VPN tunnel is up, but traffic is not able to pass through the tunnel, Wireshark (or an equivalent program) can be used to determine whether there is an encryption mismatch. A mismatch could occur for many reasons, one of the most common is the instability of an ISP link (ADSL, Cable), or it could effectively be any device in the physical connection.
The following information is required to troubleshoot the problem.
1) Take a packet sniffer trace on both FortiGates.
2) Run the 'diag vpn tunnel list' command a few times on both FortiGates when generating traffic that will pass through the tunnel.
In the following example, the error message was seen on the recipient FortiGate:
| date=2010-12-28 time=18:19:35 devname=Kosad_VPN device_id=FG300B3910600118 log_id=0101037132 type=event subtype=ipsec pri=critical vd="root" msg="IPsec ESP" action="error" rem_ip=180.87.33.2 loc_ip=121.133.8.18 rem_port=32528 loc_port=4500 out_intf="port2" cookies="88d40f65d555ccaf/05464e20e4afc835" user="N/A" group="N/A" xauth_user="N/A" xauth_group="N/A" vpn_tunnel="fortinet_0" status=esp_error error_num=Invalid ESP packet detected (HMAC validation failed). spi=c32b09f7 seq=00000012 |
This is the output of the command 'diag vpn tunnel list' on the FortiGate.
| inet ver=1 serial=2 192.168.1.205:4500->121.133.8.18:4500 lgwy=dyn tun=intf mode=auto bound_if=4 proxyid_num=1 child_num=0 refcnt=7 ilast=0 olast=0 stat: rxp=41 txp=56 rxb=4920 txb=3360 dpd: mode=active on=1 idle=5000ms retry=3 count=0 seqno=696 natt: mode=keepalive draft=32 interval=10 remote_port=4500 proxyid=P2_60C_Fortinet proto=0 sa=1 ref=2 auto_negotiate=0 serial=1 src: 0:182.40.101.0/255.255.255.0:0 dst: 0:100.100.100.0/255.255.255.0:0 SA: ref=3 options=0000000d type=00 soft=0 mtu=1428 expire=1106 replaywin=0 seqno=15 life: type=01 bytes=0/0 timeout=1777/1800 dec: spi=29a26eb6 esp=3des key=24 bf25e69df90257f64c55dda4069f01834cd0382fe4866ff2 ah=sha1 key=20 38b2600170585d2dfa646caed5bc86d920aed7ff enc: spi=c32b09f7 esp=3des key=24 0abd3c70032123c3369a6f225a385d30f0b2fb1cd9687ec8 ah=sha1 key=20 214d8e717306dffceec3760464b6e8edb436c6 |
This is the packet capture from the FortiGate:
How to verify if the original packet has been encrypted correctly?
To verify it is necessary to decrypt the ESP packet using Wireshark. Open the packet capture that is taken from initiator FortiGate using Wireshark, go to edit -> Preferences, Expand Protocol and look for ESP. Select the Check Box 'Attempt to detect/decode encrypted ESP payloads', and fill in the information for the encryption algorithm and the keys. This information can be obtained from the output of the command 'diag vpn tunnel list'.
If the packet was encrypted correctly using the correct key, then the decryption will be successful and it will be possible to see the original package as shown below:
Repeat the decryption process for the packet capture from the recipient firewall. If the decryption failed using the same key, the packet may be corrupted and the interface should then be checked for CRC or packet errors
