Description
This article describes how routing decisions work in FortiGate with or without asym routing, and with or without an auxiliary session enabled.
Scope
FortiGate Version 6 and above.
Solution
In this scenario, the traffic flows between a Client and a Server passing through two FortiGates.
The client and server are connected to their corresponding FortiGate via Port 2.
The FortiGate Server and FortiGate Client are connected to each other via two links: Port 3 < - > Port 3 and Port 4 < - > Port 4. Each link uses a specific VLAN with specific IP addresses.
Routing table:
FortiGate Client:
- Default route via Port 4.
- Specific route for server’s subnet via Port 3.
get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B – BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default
Routing table for VRF=0
S* 0.0.0.0/0 [10/0] via 10.203.12.129, port4
C 10.12.0.0/20 is directly connected, port2
S 10.186.0.0/20 [10/0] via 10.200.12.129, port3
C 10.200.0.0/20 is directly connected, port3
C 10.203.0.0/20 is directly connected, port4
FortiGate Server:
- Default route via port 3
- Specific route for client’s subnet via port 4.
get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B – BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default
Routing table for VRF=0
S* 0.0.0.0/0 [10/0] via 10.200.12.135, port3
S 10.12.0.0/20 [10/0] via 10.203.12.135, port4
C 10.186.0.0/20 is directly connected, port2
C 10.200.0.0/20 is directly connected, port3
C 10.203.0.0/20 is directly connected, port4
In this scenario, a TCP session will be generated from the client to the server. The return path used will be observed.
Test 1: Default configuration:
asymroute disable
auxiliary-session disable
FortiGate Client:
diagnose sniffer packet any 'host 10.12.12.137' 4 0 l
Using Original Sniffing Mode
interfaces=[any]
filters=[host 10.12.12.137]
2024-02-07 13:29:55.316437 port2 in 10.12.12.137.60790 -> 10.186.12.136.8080: syn 2826351306
2024-02-07 13:29:55.316801 port3 out 10.12.12.137.60790 -> 10.186.12.136.8080: syn 2826351306
2024-02-07 13:29:55.323663 port3 in 10.186.12.136.8080 -> 10.12.12.137.60790: syn 3465213974 ack 2826351307
2024-02-07 13:29:55.324960 port2 out 10.186.12.136.8080 -> 10.12.12.137.60790: syn 3465213974 ack 2826351307
2024-02-07 13:29:55.326016 port2 in 10.12.12.137.60790 -> 10.186.12.136.8080: ack 3465213975
2024-02-07 13:29:55.326037 port3 out 10.12.12.137.60790 -> 10.186.12.136.8080: ack 3465213975
2024-02-07 13:29:57.235256 port2 in 10.12.12.137.60790 -> 10.186.12.136.8080: psh 2826351307 ack 3465213975
2024-02-07 13:29:57.235311 port3 out 10.12.12.137.60790 -> 10.186.12.136.8080: psh 2826351307 ack 3465213975
2024-02-07 13:29:57.241008 port3 in 10.186.12.136.8080 -> 10.12.12.137.60790: ack 2826351316
2024-02-07 13:29:57.241039 port2 out 10.186.12.136.8080 -> 10.12.12.137.60790: ack 2826351316
2024-02-07 13:30:02.595574 port3 in 10.186.12.136.8080 -> 10.12.12.137.60790: psh 3465213975 ack 2826351316
2024-02-07 13:30:02.595632 port2 out 10.186.12.136.8080 -> 10.12.12.137.60790: psh 3465213975 ack 2826351316
2024-02-07 13:30:02.600470 port2 in 10.12.12.137.60790 -> 10.186.12.136.8080: ack 3465213984
2024-02-07 13:30:02.600504 port3 out 10.12.12.137.60790 -> 10.186.12.136.8080: ack 3465213984
2024-02-07 13:30:04.283475 port3 in 10.186.12.136.8080 -> 10.12.12.137.60790: fin 3465213984 ack 2826351316
2024-02-07 13:30:04.283533 port2 out 10.186.12.136.8080 -> 10.12.12.137.60790: fin 3465213984 ack 2826351316
2024-02-07 13:30:04.288448 port2 in 10.12.12.137.60790 -> 10.186.12.136.8080: fin 2826351316 ack 3465213985
2024-02-07 13:30:04.288473 port3 out 10.12.12.137.60790 -> 10.186.12.136.8080: fin 2826351316 ack 3465213985
2024-02-07 13:30:04.290218 port3 in 10.186.12.136.8080 -> 10.12.12.137.60790: ack 2826351317
2024-02-07 13:30:04.290246 port2 out 10.186.12.136.8080 -> 10.12.12.137.60790: ack 28263513170
FortiGate Server:
diagnose sniffer packet any 'host 10.12.12.137' 4 0 l
Using Original Sniffing Mode
interfaces=[any]
filters=[host 10.12.12.137]
2024-02-07 13:29:55.254489 port3 in 10.12.12.137.60790 -> 10.186.12.136.8080: syn 2826351306
2024-02-07 13:29:55.254849 port2 out 10.12.12.137.60790 -> 10.186.12.136.8080: syn 2826351306
2024-02-07 13:29:55.260460 port2 in 10.186.12.136.8080 -> 10.12.12.137.60790: syn 3465213974 ack 2826351307
2024-02-07 13:29:55.260577 port3 out 10.186.12.136.8080 -> 10.12.12.137.60790: syn 3465213974 ack 2826351307
2024-02-07 13:29:55.263297 port3 in 10.12.12.137.60790 -> 10.186.12.136.8080: ack 3465213975
2024-02-07 13:29:55.263325 port2 out 10.12.12.137.60790 -> 10.186.12.136.8080: ack 3465213975
2024-02-07 13:29:57.172725 port3 in 10.12.12.137.60790 -> 10.186.12.136.8080: psh 2826351307 ack 3465213975
2024-02-07 13:29:57.172782 port2 out 10.12.12.137.60790 -> 10.186.12.136.8080: psh 2826351307 ack 3465213975
2024-02-07 13:29:57.177614 port2 in 10.186.12.136.8080 -> 10.12.12.137.60790: ack 2826351316
2024-02-07 13:29:57.177638 port3 out 10.186.12.136.8080 -> 10.12.12.137.60790: ack 2826351316
2024-02-07 13:30:02.532112 port2 in 10.186.12.136.8080 -> 10.12.12.137.60790: psh 3465213975 ack 2826351316
2024-02-07 13:30:02.532169 port3 out 10.186.12.136.8080 -> 10.12.12.137.60790: psh 3465213975 ack 2826351316
2024-02-07 13:30:02.537766 port3 in 10.12.12.137.60790 -> 10.186.12.136.8080: ack 3465213984
2024-02-07 13:30:02.537795 port2 out 10.12.12.137.60790 -> 10.186.12.136.8080: ack 3465213984
2024-02-07 13:30:04.220029 port2 in 10.186.12.136.8080 -> 10.12.12.137.60790: fin 3465213984 ack 2826351316
2024-02-07 13:30:04.220086 port3 out 10.186.12.136.8080 -> 10.12.12.137.60790: fin 3465213984 ack 2826351316
2024-02-07 13:30:04.225954 port3 in 10.12.12.137.60790 -> 10.186.12.136.8080: fin 2826351316 ack 3465213985
2024-02-07 13:30:04.225982 port2 out 10.12.12.137.60790 -> 10.186.12.136.8080: fin 2826351316 ack 3465213985
2024-02-07 13:30:04.226953 port2 in 10.186.12.136.8080 -> 10.12.12.137.60790: ack 2826351317
2024-02-07 13:30:04.226970 port3 out 10.186.12.136.8080 -> 10.12.12.137.60790: ack 2826351317
The default configuration shows the FortiGate Client receives the traffic from Port 2 (Client’s port), and forwards it to Port 3. Port 3 is chosen because it has a smaller subnet route than the Server’s IP subnet.
On the FortiGate Server, the traffic comes into Port 3, and is forwarded to the server with Port 2 (the Server’s port). As the traffic is coming from Port 3, the FortiGate will keep this port to forward the return traffic.
RPF accepts the traffic because there is a feasible route (default route), even if the best route is the one related to Port 4 (Smaller subnet).
On the FortiGate Server, Port 3 will continue to be used even if it is not the one with the best route. As the flow originally comes from this port, this port will be written in the sessions table and will be conserved to send back the return traffic.
When the first packet of a session is matched in the policy table, stateful inspection adds information about the session to its session table. This session table contains the interface information.
So when subsequent packets are received for the same session, stateful inspection can determine how to handle them by looking them up in the session table (which is more efficient than in the policy table).
Stateful inspection decides to drop or allow a session and apply security features to it based on what is found in the first packet of the session. Then all subsequent packets in the same session are processed in the same way.
When the final packet in the session is processed, the session is removed from the session table.
Stateful inspection also has a session idle timeout that removes sessions from the session table that have been idle for the length of the timeout.
Test 2: Auxiliary session enabled, asymmetric routing disabled.
asymroute disable
auxiliary-session enable
In the Test 2 scenario, an auxiliary session is enabled. The remaining config is the same as in the Test 1 scenario.
FortiGate Client:
diagnose sniffer packet any 'host 10.12.12.137' 4 0 l
Using Original Sniffing Mode
interfaces=[any]
filters=[host 10.12.12.137]
2024-02-07 13:58:20.431798 port2 in 10.12.12.137.47514 -> 10.186.12.136.8080: syn 3934655152
2024-02-07 13:58:20.432128 port3 out 10.12.12.137.47514 -> 10.186.12.136.8080: syn 3934655152
2024-02-07 13:58:20.441073 port4 in 10.186.12.136.8080 -> 10.12.12.137.47514: syn 3600518869 ack 3934655153
2024-02-07 13:58:20.444819 port2 out 10.186.12.136.8080 -> 10.12.12.137.47514: syn 3600518869 ack 3934655153
2024-02-07 13:58:20.445675 port2 in 10.12.12.137.47514 -> 10.186.12.136.8080: ack 3600518870
2024-02-07 13:58:20.445716 port3 out 10.12.12.137.47514 -> 10.186.12.136.8080: ack 3600518870
2024-02-07 13:58:24.571978 port2 in 10.12.12.137.47514 -> 10.186.12.136.8080: psh 3934655153 ack 3600518870
2024-02-07 13:58:24.572027 port3 out 10.12.12.137.47514 -> 10.186.12.136.8080: psh 3934655153 ack 3600518870
2024-02-07 13:58:24.577547 port4 in 10.186.12.136.8080 -> 10.12.12.137.47514: ack 3934655169
2024-02-07 13:58:24.577603 port2 out 10.186.12.136.8080 -> 10.12.12.137.47514: ack 3934655169
2024-02-07 13:58:31.563508 port4 in 10.186.12.136.8080 -> 10.12.12.137.47514: psh 3600518870 ack 3934655169
2024-02-07 13:58:31.563566 port2 out 10.186.12.136.8080 -> 10.12.12.137.47514: psh 3600518870 ack 3934655169
2024-02-07 13:58:31.568668 port2 in 10.12.12.137.47514 -> 10.186.12.136.8080: ack 3600518879
2024-02-07 13:58:31.568729 port3 out 10.12.12.137.47514 -> 10.186.12.136.8080: ack 3600518879
2024-02-07 13:58:32.972624 port4 in 10.186.12.136.8080 -> 10.12.12.137.47514: fin 3600518879 ack 3934655169
2024-02-07 13:58:32.972731 port2 out 10.186.12.136.8080 -> 10.12.12.137.47514: fin 3600518879 ack 3934655169
2024-02-07 13:58:32.977962 port2 in 10.12.12.137.47514 -> 10.186.12.136.8080: fin 3934655169 ack 3600518880
2024-02-07 13:58:32.978018 port3 out 10.12.12.137.47514 -> 10.186.12.136.8080: fin 3934655169 ack 3600518880
2024-02-07 13:58:32.979398 port4 in 10.186.12.136.8080 -> 10.12.12.137.47514: ack 3934655170
2024-02-07 13:58:32.979450 port2 out 10.186.12.136.8080 -> 10.12.12.137.47514: ack 3934655170
For the first packets in Test 1, the Client uses Port 2 to reach the FortiGate. The FortiGate then uses Port 3 to reach the FortiGate Server.
There is a different behavior for the received SYN-ACK; it comes from Port 4, which was received on Port 3 with the default configuration. It will be seen more in detail with the FortiGate Server sniffer.
FortiGate Server:
diagnose sniffer packet any 'host 10.12.12.137' 4 0 l
Using Original Sniffing Mode
interfaces=[any]
filters=[host 10.12.12.137]
2024-02-07 13:58:20.369920 port3 in 10.12.12.137.47514 -> 10.186.12.136.8080: syn 3934655152
2024-02-07 13:58:20.375192 port2 out 10.12.12.137.47514 -> 10.186.12.136.8080: syn 3934655152
2024-02-07 13:58:20.376503 port2 in 10.186.12.136.8080 -> 10.12.12.137.47514: syn 3600518869 ack 3934655153
2024-02-07 13:58:20.377613 port4 out 10.186.12.136.8080 -> 10.12.12.137.47514: syn 3600518869 ack 3934655153
2024-02-07 13:58:20.382984 port3 in 10.12.12.137.47514 -> 10.186.12.136.8080: ack 3600518870
2024-02-07 13:58:20.383007 port2 out 10.12.12.137.47514 -> 10.186.12.136.8080: ack 3600518870
2024-02-07 13:58:24.509496 port3 in 10.12.12.137.47514 -> 10.186.12.136.8080: psh 3934655153 ack 3600518870
2024-02-07 13:58:24.509547 port2 out 10.12.12.137.47514 -> 10.186.12.136.8080: psh 3934655153 ack 3600518870
2024-02-07 13:58:24.514365 port2 in 10.186.12.136.8080 -> 10.12.12.137.47514: ack 3934655169
2024-02-07 13:58:24.514391 port4 out 10.186.12.136.8080 -> 10.12.12.137.47514: ack 3934655169
2024-02-07 13:58:31.500186 port2 in 10.186.12.136.8080 -> 10.12.12.137.47514: psh 3600518870 ack 3934655169
2024-02-07 13:58:31.500235 port4 out 10.186.12.136.8080 -> 10.12.12.137.47514: psh 3600518870 ack 3934655169
2024-02-07 13:58:31.505929 port3 in 10.12.12.137.47514 -> 10.186.12.136.8080: ack 3600518879
2024-02-07 13:58:31.505954 port2 out 10.12.12.137.47514 -> 10.186.12.136.8080: ack 3600518879
2024-02-07 13:58:32.909338 port2 in 10.186.12.136.8080 -> 10.12.12.137.47514: fin 3600518879 ack 3934655169
2024-02-07 13:58:32.909394 port4 out 10.186.12.136.8080 -> 10.12.12.137.47514: fin 3600518879 ack 3934655169
2024-02-07 13:58:32.915356 port3 in 10.12.12.137.47514 -> 10.186.12.136.8080: fin 3934655169 ack 3600518880
2024-02-07 13:58:32.915380 port2 out 10.12.12.137.47514 -> 10.186.12.136.8080: fin 3934655169 ack 3600518880
2024-02-07 13:58:32.916136 port2 in 10.186.12.136.8080 -> 10.12.12.137.47514: ack 3934655170
2024-02-07 13:58:32.916150 port4 out 10.186.12.136.8080 -> 10.12.12.137.47514: ack 3934655170
For the first packets in Test 1, the FortiGate Server receives the traffic on Port 3. It is forwarded to the client on Port 2.
For the SYN-ACK, the behavior is different. The FortiGate Server sends the traffic towards the FortiGate Client through Port 4. Port 4 is the best route to reach the FortiGate Client from the FortiGate Server.
The reply to the client egresses on the best route in the routing table:
- If the best route is through Port 3, it will egress on Port 3.
- If the best route is through Port 4, it will egress on Port 4.
No matter where the initial packets are coming from.
Enabling auxiliary sessions allows to control where the return traffic should go. The routing table will be taken into account and not the original incoming interface.
Moreover, unlike with asymmetric routing enabled, the FortiGate remains a stateful Firewall, and all UTM features can be enabled.
Test 3: Asymmetric routing enable.
asymroute enable
auxiliary-session disable
FortiGate Client:
diagnose sniffer packet any 'host 10.12.12.137' 4 0 l
Using Original Sniffing Mode
interfaces=[any]
filters=[host 10.12.12.137]
2024-02-07 15:02:24.693139 port2 in 10.12.12.137.35696 -> 10.186.12.136.8080: syn 3191568935
2024-02-07 15:02:24.693484 port3 out 10.12.12.137.35696 -> 10.186.12.136.8080: syn 3191568935
2024-02-07 15:02:24.701203 port4 in 10.186.12.136.8080 -> 10.12.12.137.35696: syn 3122815364 ack 3191568936
2024-02-07 15:02:24.702674 port2 out 10.186.12.136.8080 -> 10.12.12.137.35696: syn 3122815364 ack 3191568936
2024-02-07 15:02:24.705191 port2 in 10.12.12.137.35696 -> 10.186.12.136.8080: ack 3122815365
2024-02-07 15:02:24.705257 port3 out 10.12.12.137.35696 -> 10.186.12.136.8080: ack 3122815365
2024-02-07 15:02:27.050916 port2 in 10.12.12.137.35696 -> 10.186.12.136.8080: psh 3191568936 ack 3122815365
2024-02-07 15:02:27.050975 port3 out 10.12.12.137.35696 -> 10.186.12.136.8080: psh 3191568936 ack 3122815365
2024-02-07 15:02:27.056795 port4 in 10.186.12.136.8080 -> 10.12.12.137.35696: ack 3191568947
2024-02-07 15:02:27.056854 port2 out 10.186.12.136.8080 -> 10.12.12.137.35696: ack 3191568947
2024-02-07 15:02:31.323407 port4 in 10.186.12.136.8080 -> 10.12.12.137.35696: psh 3122815365 ack 3191568947
2024-02-07 15:02:31.323487 port2 out 10.186.12.136.8080 -> 10.12.12.137.35696: psh 3122815365 ack 3191568947
2024-02-07 15:02:31.328237 port2 in 10.12.12.137.35696 -> 10.186.12.136.8080: ack 3122815376
2024-02-07 15:02:31.328280 port3 out 10.12.12.137.35696 -> 10.186.12.136.8080: ack 3122815376
2024-02-07 15:02:31.916646 port4 in 10.186.12.136.8080 -> 10.12.12.137.35696: fin 3122815376 ack 3191568947
2024-02-07 15:02:31.916735 port2 out 10.186.12.136.8080 -> 10.12.12.137.35696: fin 3122815376 ack 3191568947
2024-02-07 15:02:31.921284 port2 in 10.12.12.137.35696 -> 10.186.12.136.8080: fin 3191568947 ack 3122815377
2024-02-07 15:02:31.921329 port3 out 10.12.12.137.35696 -> 10.186.12.136.8080: fin 3191568947 ack 3122815377
2024-02-07 15:02:31.922105 port4 in 10.186.12.136.8080 -> 10.12.12.137.35696: ack 3191568948
2024-02-07 15:02:31.922141 port2 out 10.186.12.136.8080 -> 10.12.12.137.35696: ack 3191568948
The first packet is similar to Test 1 and Test 2.
The SYN-ACK packet comes from Port 4. The behavior is the same as that with an auxiliary session enabled.
The main difference will be related to Hardware Acceleration. Some features do not work when asymmetric routing is enabled.
FortiGate Server:
diagnose sniffer packet any 'host 10.12.12.137' 4 0 l
Using Original Sniffing Mode
interfaces=[any]
filters=[host 10.12.12.137]
2024-02-07 15:02:24.631209 port3 in 10.12.12.137.35696 -> 10.186.12.136.8080: syn 3191568935
2024-02-07 15:02:24.636334 port2 out 10.12.12.137.35696 -> 10.186.12.136.8080: syn 3191568935
2024-02-07 15:02:24.637737 port2 in 10.186.12.136.8080 -> 10.12.12.137.35696: syn 3122815364 ack 3191568936
2024-02-07 15:02:24.637821 port4 out 10.186.12.136.8080 -> 10.12.12.137.35696: syn 3122815364 ack 3191568936
2024-02-07 15:02:24.643489 port3 in 10.12.12.137.35696 -> 10.186.12.136.8080: ack 3122815365
2024-02-07 15:02:24.643505 port2 out 10.12.12.137.35696 -> 10.186.12.136.8080: ack 3122815365
2024-02-07 15:02:26.988477 port3 in 10.12.12.137.35696 -> 10.186.12.136.8080: psh 3191568936 ack 3122815365
2024-02-07 15:02:26.988534 port2 out 10.12.12.137.35696 -> 10.186.12.136.8080: psh 3191568936 ack 3122815365
2024-02-07 15:02:26.993545 port2 in 10.186.12.136.8080 -> 10.12.12.137.35696: ack 3191568947
2024-02-07 15:02:26.993585 port4 out 10.186.12.136.8080 -> 10.12.12.137.35696: ack 3191568947
2024-02-07 15:02:31.259965 port2 in 10.186.12.136.8080 -> 10.12.12.137.35696: psh 3122815365 ack 3191568947
2024-02-07 15:02:31.260020 port4 out 10.186.12.136.8080 -> 10.12.12.137.35696: psh 3122815365 ack 3191568947
2024-02-07 15:02:31.265610 port3 in 10.12.12.137.35696 -> 10.186.12.136.8080: ack 3122815376
2024-02-07 15:02:31.265633 port2 out 10.12.12.137.35696 -> 10.186.12.136.8080: ack 3122815376
2024-02-07 15:02:31.853314 port2 in 10.186.12.136.8080 -> 10.12.12.137.35696: fin 3122815376 ack 3191568947
2024-02-07 15:02:31.853367 port4 out 10.186.12.136.8080 -> 10.12.12.137.35696: fin 3122815376 ack 3191568947
2024-02-07 15:02:31.858599 port3 in 10.12.12.137.35696 -> 10.186.12.136.8080: fin 3191568947 ack 3122815377
2024-02-07 15:02:31.858622 port2 out 10.12.12.137.35696 -> 10.186.12.136.8080: fin 3191568947 ack 3122815377
2024-02-07 15:02:31.859028 port2 in 10.186.12.136.8080 -> 10.12.12.137.35696: ack 3191568948
2024-02-07 15:02:31.859041 port4 out 10.186.12.136.8080 -> 10.12.12.137.35696: ack 3191568948
The behavior of enabling the asym routing in a FortiGate:
Reply Traffic will choose the best route/interface to forward traffic rather than using the same incoming interface ('sticky' behavior).
As a result, the FortiGate checks the routing table to forward the reply, regardless of which interface the packets come from.
Note that if asymmetric routing is enabled, antivirus and intrusion prevention systems will not be effective.
The FortiGate will not detect connections and will process each packet individually through the CPU. This could have an impact on the performance.
FortiGate will behave like a stateful firewall and will not take advantage of the hardware acceleration.
Enabling the asymmetric routing changes how the FortiGate handles the return traffic. Asymmetric routing can affect performance and security behavior.
A better option is to consider using auxiliary-sessions rather than asymmetric routing.
