Technical Tip: Rebuilding an HA cluster

tlegentil_FTNT · ‎09-09-2009

Description

This article provides some simple steps to follow where an HA cluster may have to be rebuilt in order to ensure basic HA operation.

Scope

FortiGate.

Solution

In cases where Administrators wish to build a HA cluster or try to recover from a synchronization issue.

Note:

It is recommended to perform these steps locally as cabling should be disconnected and reconnected to ensure a smooth rebuild. Be sure to perform these steps during a maintenance window per best practices.

Preparing for the rebuild:

Back up the primary unit config. If obtaining support from Fortinet, please provide the configuration so the Engineer can provide specific guidance.
Be careful to remove the cluster's data connections first, monitored interface cables next, and HA heartbeat cables last.
Disconnect the backup unit from the cluster. Ensure the unit is not connected to the network in any way to prevent both FortiGates from acting as primaries (split-brain).
Power off the backup by connecting to its console port and issuing "exec shutdown".
At CLI reset the backup to factory defaults: "exec factory-reset". It will take 5-10 minutes to reboot\

From the Master unit config copy the HA settings. Some of the most critical parameters are:
a. group-id
b. group-name
c. password <----- Check note below.
d. unit priority
e. mode
f. hbdev (heartbeat interface/device)
g. monitored interfaces ("monitor")

To show the settings at CLI: show system ha

To check the HA settings using the config file: Search for 'config system ha'

Example copied config:

config system ha
    set group-id 33
    set group-name "haCluster"
    set mode a-p
    set password ENC bi+kLsLH7Z8Gxyw4P/+5eIE2PoWs/Cp/aI+2qtJjlwdhP2ckfK4AFc45yKJTak9M7x1OsXtixBDCz70Uru/238zWqbXnobuuLIWqCM7udaGpWwgaXWoDi8rNPegNVtZ4yIbC5xA7T6ZwCje4/+SvKcMQ8R5AjiEskIZb3fNMhDVRbHiyGKsvFKsx1iy/vpP1OvFSUg==
    set hbdev "wan2" 50
    set session-pickup enable
    set ha-mgmt-status enable
    set ha-mgmt-interface "internal9"
    set override enable
    set priority 255
    set monitor "wan1"
end

Set the following on the new unit via console:

config system global
set hostname <secondary_unit>
end

Configure below only if there is a dedicated management interface :

config system interface
edit <mgmt-interface>
set ip <dedicated secondary_unit ip> <subnet mask>
end

Note:

Be sure to also copy the line that contains the cluster password. Paste the HA settings into a text document. If the cluster password is lost or forgotten, it can be changed on all cluster units. Change it from GUI on primary and paste the same on the above configuration in the text editor. (When the primary password changed, a few packets would be dropped or sessions would disconnected)

Rebuilding the backup:

- If override is enabled like the above config, it is recommended to set the priority below that of the primary unit. This will avoid any service interruptions when the backup is added to the cluster. For this example, we will set priority to 254 since the primary is 255.

- On the backup unit at the console paste the HA settings from the text editor.

Note:

Assuming override is disabled, whichever unit has the highest uptime will become the new primary unit. Ensure the primary unit in production has a higher uptime before connecting the backup if override is disabled (which is the default). To check: 'get sys perf stat | grep Uptime'.
- Reconnect the Slave unit(s). Connect the HA heartbeat cables first, monitored interface cables next, and data connections last when re-connecting the slave device. Note that it may reboot once when synchronizing.
- Repeat these steps for as many cluster members as are needed.

Once this is done and the cluster has been formed, the primary unit configuration will be synchronized to the backup devices. This process takes 5-20 minutes depending on the size of the configuration and how many cluster members exist. To ensure the cluster is fully synchronized, follow the checksum article referenced below.