Description
This article describes how to prevent SD-WAN member interfaces forwarding traffic gateway change due to ICMP-Redirect on the connected network unit.
Solution
SCENARIO.
FortiGate has SD-WAN with WAN1 & WAN2 member interfaces.
WAN1 has direct internet link to the ISP with Public IP address 169.254.1.0/30.
Note: 169.254.X.X Link-Local IP used as Public IP just for illustration.
WAN2 is connected to the internet via internal Layer-3 switch and ROUTER. They all are in the same network 172.17.160.16/29.
SD-WAN member interface WAN2 is configured with Layer-3 switch (172.17.160.17) as gateway. In turn, Layer-3 switch is configured with ROUTER (Which is connected to the ISP).
SD-WAN CONFIGURATION:
#config system virtual-wan-link
set status enable
config members
edit 1
set interface "WAN2"
set gateway 172.17.160.17
next
edit 2
set interface "WAN1"
set gateway 169.254.1.2
next
end
config health-check
edit "PING"
set server "8.8.8.8"
set members 1 2
next
end
#config router static
edit 1
set distance 1
set virtual-wan-link enable
next
end
Health-check status for WAN1 i.e Seq(2) is dead and WAN2 i.e Seq(1) is alive.
FORTIGATE# diag sys virtual-wan-link health-check
Health Check(PING):
Seq(2): state(dead), packet-loss(0.000%) sla_map=0x0
Seq(1): state(alive), packet-loss(0.000%) latency(11.977), jitter(0.681) sla_map=0x0
Correct gateways were displayed against WAN1 and WAN2 and can able to ping both the gateway IP’s.
FORTIGATE# diagnose ip address list
IP=172.17.160.21->172.17.160.21/255.255.255.248 index=35 devname=WAN2
IP=169.254.1.1->169.254.1.1/255.255.255.252 index=36 devname=WAN1
FORTIGATE# diagnose ip route list
tab=254 vf=0 scope=0 type=1 proto=17 prio=0 169.254.1.1/255.255.255.255/0->8.8.8.8/32 pref=0.0.0.0 gwy=169.254.1.2 dev=36(WAN1)
tab=254 vf=0 scope=0 type=1 proto=17 prio=0 172.17.160.21/255.255.255.255/0->8.8.8.8/32 pref=0.0.0.0 gwy=172.17.160.17 dev=35(WAN2)
However, the rtcache is showing the different gateway than the gateway configured in the SD-WAN member interface configuration.
- WAN2 Gateway has been changed from 172.17.160.17 to 172.17.160.20 due to ICMP redirect received from L3 Switch.
- WAN1 Gateway has been changed from 169.254.1.2 to 172.17.160.20
FORTIGATE# diag ip rtcache list
family=02 tab=254 vrf=0 vf=0 type=01 tos=0 flag=00040200
172.17.160.21@0->8.8.8.8@35(WAN2) gwy=172.17.160.20 prefsrc=0.0.0.0
ci: ref=0 lastused=0 expire=0 err=00000000 used=22895 br=0 pmtu=1500
family=02 tab=254 vrf=0 vf=0 type=01 tos=0 flag=00000200
169.254.1.1@0->8.8.8.8@36(WAN1) gwy=172.17.160.20 prefsrc=0.0.0.0
ci: ref=4 lastused=0 expire=0 err=00000000 used=27472 br=0 pmtu=1500
EXPLANATION ABOUT WHY THE GATEWAY HAS BEEN CHANGED FOR FORWARDING TRAFFIC.
For WAN2, when local traffic originated by the health-check for destination 8.8.8.8 with 172.17.160.21 as a source
Gateway Layer-3 Switch with IP address 172.17.160.17 receives data packet from a 172.17.160.21 on a network to which it is attached.
The gateway, Layer-3 Switch, checks its routing table and obtains the address 172.17.160.20 of the next gateway, ROUTER, on the route to data packet's destination network 8.8.8.8.
As ROUTER (172.17.160.20/29) and the FortiGate identified by the source address (i.e 172.17.160.21/29) of IP packet are on the same network, Layer-3 switch sends ICMP Redirect message to the FortiGate on WAN2.
The ICMP Redirect message advises the FortiGate to send its traffic for Network (here, 8.8.8.8) directly to gateway ROUTER as this is a shorter path to the destination. Hence, route cache (rtcache) list is displaying the gateway as 172.17.160.20 for WAN2.
As WAN1 and WAN2 are the member interface of SD-WAN, gateway change has been seen for WAN1 as well.
PACKET CAPTURED ON FORTIGATE ILLUSTRATING THE ICMP REDIRECT MESSAGE RECEIVED FROM LAYER-3 SWITCH.
Due to gateway change, WAN2 traffic is not impacted as new gateway belongs to the same network and hence health-check for the WAN2 is UP.
However, WAN1 gateway changed to the private IP 172.17.160.20 and hence WAN1 health-check is down.
Related document.ICMP Redirects are enabled by default on Layer 3 interfaces.
When ICMP Redirect messages are generated: Layer3 device generates ICMP Redirect message back to the source of data packet, if data packet is to be forwarded out the Layer 3 interface on which this packet is received.
See the below options to prevent this issue.1) Disable icmp redirect on the Layer-3 switch (Can be any other Layer-3 device like router).2) Disable accepting and sending ICMP redirects on Fortigate WAN interfaces.
# config system interface
edit "WAN2 "
set icmp-accept-redirect disable
set icmp-send-redirect disable
next
edit "WAN1”
set icmp-accept-redirect disable
set icmp-send-redirect disable
end
RESULT.
FORTIGATE# diag ip rtcache list
family=02 tab=254 vrf=0 vf=0 type=01 tos=0 flag=00040200
172.17.160.21@0->8.8.8.8@35(WAN2) gwy=172.17.160.17 prefsrc=0.0.0.0
ci: ref=0 lastused=0 expire=0 err=00000000 used=22895 br=0 pmtu=1500
family=02 tab=254 vrf=0 vf=0 type=01 tos=0 flag=00000200
169.254.1.1@0->8.8.8.8@36(WAN1) gwy=169.254.1.2 prefsrc=0.0.0.0
ci: ref=4 lastused=0 expire=0 err=00000000 used=27472 br=0 pmtu=1500
FORTIGATE# diag sys virtual-wan-link health-check
Health Check(PING):
Seq(2): state(alive), packet-loss(0.000%) latency(10.567), jitter(0.230) sla_map=0x0
Seq(1): state(alive), packet-loss(0.000%) latency(11.977), jitter(0.681) sla_map=0x0
As explained above, the ICMP redirect packets which are received from one interface can change the gateways of other routing caches which have the same destination but different egress interfaces.This behavior would make FortiGate to send a packet from an interface to an invalid gateway which is not desired behavior hence it has been fixed in v6.2.8, v6.4.3 and later versions.
