Technical Tip: Behavior of Virtual MAC usage with VDOM-partioning

[Anthony_E]

Description
This article describes the changes in VMAC assignment in a HA cluster depending on the implementation.

Solution
When a cluster is formed, the interfaces are assigned a VMAC.
This VMAC helps in seamless transition of traffic to the active box following a failover.
When the cluster starts up, after a fail-over, the primary unit sends gratuitous ARP packets to update the switches connected to the cluster interfaces with the virtual MAC address.
The switches update the MAC forwarding tables with this MAC address.
As a result, the switches direct all network traffic to the primary unit.
Depending on the cluster configuration, the primary unit either processes this network traffic itself or load balances the network traffic among all of the cluster units.

Note: In normal scenario, GARP is intended to change the MAC table of the switch and do not intend to do any changes on the ARP table.
After a fail-over, because the new primary unit has the same IP addresses and MAC addresses (VMAC) as the failed primary unit, once the switches updated the MAC forwarding tables no information about the fail over needs to be communicated to other network units.

However, with the VDOM portioning enabled, the behavior changes.

Scenario 1: HA in active – passive mode with no VDOM-partitioning.

On Primary.

Primary # diag hardware deviceinfo nic port2 | grep addr
Hwaddr            : 00:09:0f:09:75:01                              <----- VMAC.
Permanent Hwaddr    : 00:6e:65:75:2c:02
Primary # diag hardware deviceinfo nic port3 | grep addr
Hwaddr            : 00:09:0f:09:75:02                              <----- VMAC.
Permanent Hwaddr    : 00:6e:65:75:2c:03

On Secondary.

Secondary # diag hardware deviceinfo nic port2 | grep addr
Hwaddr            : 00:6e:65:75:10:02                              <----- Retains original MAC.
Permanent Hwaddr    : 00:6e:65:75:10:02
Secondary # diag hardware deviceinfo nic port3 | grep addr
Hwaddr            : 00:6e:65:75:10:03
Permanent Hwaddr    : 00:6e:65:75:10:03

After Failover, the VMAC ‘00:09:0f:09:75:02’ shifts to secondary box which becomes active.

Primary # diag hardware deviceinfo nic port2 | grep addr
Hwaddr            : 00:6e:65:75:2c:02
Permanent Hwaddr     : 00:6e:65:75:2c:02
Primary # diag hardware deviceinfo nic port3 | grep addr
Hwaddr            : 00:6e:65:75:2c:03
Permanent Hwaddr    : 00:6e:65:75:2c:03
 
Secondary # diag hardware deviceinfo nic port2 | grep addr
Hwaddr            : 00:09:0f:09:75:01
Permanent Hwaddr    : 00:6e:65:75:10:02
Secondary # diag hardware deviceinfo nic port3 | grep addr
Hwaddr            : 00:09:0f:09:75:02
Permanent Hwaddr    : 00:6e:65:75:10:03

Scenario 2: HA in active – passive mode with VDOM-partitioning

2.1) When interfaces are part of VDOMs and not root VDOM.

The VMACs are available on units that have the corresponding VDOMs active.
In the below example, VDOM1 is active on Primary and VDOM2 is active on Secondary.
VDOM1 has PORT2 and VDOM2 has Port3.

Primary (global)  # diag hardware deviceinfo nic port2 | grep addr
Hwaddr            : 00:09:0f:09:75:01                              <----- VMAC.
Permanent Hwaddr    : 00:6e:65:75:2c:02
Primary (global)  # diag hardware deviceinfo nic port3 | grep addr
Hwaddr            : 00:6e:65:75:2c:03                              <----- Not VMAC.
Permanent Hwaddr    : 00:6e:65:75:2c:03
 
Secondary (global) # diag hardware deviceinfo nic port2 | grep addr
Hwaddr            : 00:6e:65:75:10:02                              <----- Not VMAC.
Permanent Hwaddr    : 00:6e:65:75:10:02
Secondary (global) # diag hardware deviceinfo nic port3 | grep addr
Hwaddr            : 00:09:0f:09:75:82                              <----- VMAC.
Permanent Hwaddr    : 00:6e:65:75:10:03

In case of failovers, the VMAC passes over to the active unit.
VMAC is associated with Vcluster (as its calculation is also taking vcluster as a parameter).
 
2.2) When Port2 is part of root VDOM and VLAN interfaces are part of VDOMs.

In the below output, Primary has no VDOMs active except the root. 

In this case, with NP6 Acceleration, VLAN interface can not have different MAC than the Physical interface it belongs to. Therefore VDOMs(active on secondary) will have the physical MAC address to communicate outside. 

However for ARP requests, the VMAC will still be used.

Hence there will be inconsistency between the ARP tables and MAC tables of the upstread devices and the source MAC received by FortiGate. 

It is important for the upstream device to respond based on the ARP table (using VMAC) instead of the Physical interface MAC received in source MAC field, otherwise FortiGate kernel will drop the packet.

Primary (global)   # diag hardware deviceinfo nic port2 | grep addr

Hwaddr            : 00:09:0f:09:75:01

Permanent Hwaddr    : 00:6e:65:75:2c:02

Secondary (global) # diag hardware deviceinfo nic port2 | grep addr

Hwaddr            : 00:6e:65:75:10:02                              <-----

Permanent Hwaddr    : 00:6e:65:75:10:02 

This behavior is not observed in case of NP6 is not available or manually disabled, in this cases VMAC will be always used to communicate.

Conclusion.

 VMAC is tied to interface and associated with Vcluster.

Unless the physical interface is not part of root interface, the VMAC is assigned to the interface and will fail-over seamless with no dependency on upstream network units to update the ARP tables.

However, If the physical interface is in the root VDOM(which is always active on the Vcluster1), Due to NP6 limitation, VDOMs in Vcluster2 will use the physical MAC address to forward traffic which can lead to some MAC address inconsistencies.