Description
This article describes specific behavior on selecting routes.
The result can then differ between offloaded and non-offloaded sessions and cause undesired behavior.
Solution
Whenever the kernel's routing table is updated, all sessions (even if unrelated) will be triggered to evaluate the routing result again.
This is to simply re-evaluate the routing based on current/updated information.
There is however a behavior difference in NP offloaded sessions and non-offloaded session when making the decisions.
For an offloaded session, FortiGate will, in this order:
1) check if the route is still valid on the known output interface, there is still a valid route to that destination with the original output interface and the gateway has not changed, this session will NOT be marked dirty. Basically no change is done.
2) Then if the route does not exist or has changed, the session will be marked dirty and the known original output interface will be dropped on the session.
This will also clear the session from NPU (if offloaded).
The next packet will be checked against the routing table again to find the best match.
For non-offloaded sessions, FortiGate will skip 1) - re-evaluate and keep the session and will just execute 2) - drop interface info and re-evaluate the first packet after routing change.
For the non-offloaded session though, if the routing updates affect live traffic, the result could be that traffic is suddenly routed differently.
This can affect the traffic in an undesirable way if the new, undesired routing information leads to a dead end for the particular traffic.
Practically it might look like the connection or session has been dropped at that point. This can happen on existing packets in original direction, but also in the same way on return packets.
This is visible in the flow trace (as 'find a route') as the routing decision suddenly changes, but also is visible on the session list (truncated output).
In this first snippet the original routing and its interfaces (43<>58) are seen. Note the serial as well as it is the session identifier in the session list and flow trace.
This session is not offloaded.
session info: proto=6 proto_state=01 duration=39 expire=3599 timeout=3600 flags=00000000 sockflag=00000000 sockport=0 av_idx=0 use=4
state=may_dirty synced
orgin->sink: org pre->post, reply pre->post dev=43->58/58->43 gwy=172.16.100.10/192.168.100.1
hook=pre dir=org act=noop 10.0.100.127:59843->172.16.100.10:443(0.0.0.0:0)
hook=post dir=reply act=noop 172.16.100.10:443->10.0.100.127:59843(0.0.0.0:0)
pos/(before,after) 0/(0,0), 0/(0,0)
dst_mac=00:09:0f:09:00:00
serial=00296968 tos=ff/ff app_list=0 app=0 url_cat=0
npu_state=0x040000
no_ofld_reason: non-npu-intf
The second session snipped shows the session got marked as dirty now and the interfaces (dev) have changed according to the new routing information.
Unitschanged from 43<>58 to 60<>0 whereas there are expected to stay the same.
session info: proto=6 proto_state=01 duration=54 expire=3599 timeout=3600 flags=00000000 sockflag=00000000 sockport=0 av_idx=0 use=7
state=dirty may_dirty npu synced
orgin->sink: org pre->post, reply pre->post dev=60->0/0->60 gwy=0.0.0.0/192.168.254.124
hook=pre dir=org act=noop 10.0.100.127:59843->172.16.100.10:443(0.0.0.0:0)
hook=post dir=reply act=noop 172.16.100.10:443->10.0.100.127:59843(0.0.0.0:0)
pos/(before,after) 0/(0,0), 0/(0,0)
dst_mac=00:09:0f:09:00:00
serial=00296968 tos=ff/ff app_list=0 app=0 url_cat=0
npu_state=0x040000
npu info: flag=0x00/0x00, offload=0/0, ips_offload=0/0, epid=0/0, ipid=0/0, vlan=0x0000/0x0000
vlifid=0/0, vtag_in=0x0000/0x0000 in_npu=0/0, out_npu=0/0, fwd_en=0/0, qid=0/0
no_ofld_reason: dirty
If offloading is used on the interfaces, there is no such behavior expected.
In some cases, the capability of offloading by network design or hardware will not be granted.
Work around this eliminating the 'disturbing' routes will be possible, redesign to use offloading interfaces instead or eventually use NAT between units/VDOMs to influence the routing decisions.
See the following guide about hardware acceleration.
https://docs.fortinet.com/document/fortigate/6.4.0/hardware-acceleration/448300/hardware-acceleratio...
