Technical Note: Verification of SLBC status before an upgrade

[gfranceschi]

Description

Prior to launching the upgrade of FortiOS, verify the status of the SLBC:

• CPU/memory
• HA synchronization
• Interface status
• Forced master/slave FortiController feature status 
  

On FortiControllers:

• get sys statget sys performance
• get load-balance status
• diagnose system ha status
• diagnose sys ha showcsum 
  

On each FortiGate:

• config global
• get sys status
• get system perf status
• diagnose sys confsync status
• diagnose sys confsync showcsum

Solution

The SLBC cluster is formed with 2 chassis and each contains one FortiController in slot1, and two worker blades in slots 3 and 13.

On FortiController

get sys stat

==>  Check which unit is master or slave

et sys performance

==>  Check the CPU and memory usage

FT-SLOT1 # get load-balance status

ELBC Master Blade: slot-3

Confsync Master Blade: slot-3

Blades:

Working: 2 [ 2 Active 0 Standby]
Ready: 0 [ 0 Active 0 Standby]
Dead: 0 [ 0 Active 0 Standby]
Total: 2 [ 2 Active 0 Standby]
Slot 3: Status:Working  Function:Active
Link: Base: Up Fabric: Up
Heartbeat: Management: Good Data: Good
Status Message:"Running"
Slot 13: Status:Working  Function:Active
Link: Base: Up Fabric: Up
Heartbeat: Management: Good Data: Good
Status Message:"Running"

==>  All blades should be in "Working" state with a status message "Running"

==>  Link Base and Fabric should be UP

FT-SLOT1 # diag sys ha status

mode: a-p
minimize chassis failover: 1
FT513Cxxxxxxxxxx(FT513Cxxxxxxxxxx), Master(priority=0), ip=169.254.128.82, uptime=440.85, chassis=1(1)
slot: 1
sync: conf_sync=1, elbc_sync=1
session: total=60,  session_sync=in sync
state: gateway_die=0, worker_failure=0/2, lag=(total/good/down/bad-score)=0/0/0/0,
intf_state=(port up)=2, force-state(0:none)
hbdevs: local_interface=  b1      best=yes
local_interface= b2                       best=no
FT513Cyyyyyyyyyy(FT513Cyyyyyyyyyy), Slave(priority=1), ip=169.254.128.83, uptime=125.97, chassis=2(1)
slot: 1
sync: conf_sync=1, elbc_sync=1, conn=3(connected)
session: total=52,  session_sync=in sync
state: gateway_die=0, worker_failure=0/2, lag=(total/good/down/bad-score)=0/0/0/0,
intf_state=(port up)=1, force-state(0:none)
hbdevs: local_interface= b1 last_hb_time= 2080.16 status=alive
local_interface= b2 last_hb_time= 0.00  status=dead

==>  Both FortiControllers from slot1 in chassis 1 and chassis 2 are synchronized

==>  There is no worker blade failure "worker_failure=0/2"

==>  Base communication is active

==>  There is no issue on lag "lag=(total/good/down/bad-score)=0/0/0/0"

==>  FortiController is not forced to be master or slave "force-state(0:none)"

diagnose sys ha showcsum

==>  The outputs of this command are equal on the master and slave FortiControllers

On FortiGate

config global
get sys status

==>  Identify the blade

get system perf status

==>  Check the CPU and memory

FGT-SLOT3 (global) # diagnose sys confsync status
ELBC: svcgrp_id=1, slot_id=3
ELBC HB devs:
        elbc-ctrl/1: active=1, hb_count=3069
        elbc-ctrl/2: active=0, hb_count=0
ELBC mgmt devs:
        elbc-base-ctrl: mgmtip_set=1
zone: self_idx:0, master_idx:0
FG-5KDxxxxxxxxxx, Master, uptime=3068.54, priority=0, slot_id=1:3, idx=0, in_sync=1
FG-5KDyyyyyyyyyy, Slave, uptime=438.83, priority=1, slot_id=1:13, idx=1, in_sync=0
        elbc-base-ctrl: state=3(connected), ip=169.254.1.13, last_hb_time=3158.54, hb_nr=2009

==>  All slave worker blades have to be synchronized (in_sync =1)

==>  The blade marked "in_sync=0"  is not in_synch and not ready for the upgrade

==>  The communication between the blades and the FortiController in slot 1 is active "elbc-ctrl/1: active=1"

diagnose sys confsync showcsum

==>  The output of this command should be equal on all worker blades

If the communication is not established between the FortiController and the worker blades, or if a worker blade is in failed state, the upgrade will not take place correctly.  Some units maybe not upgraded.

To upgrade the FortiController, the upgrade should be done from the FortiController master.

To upgrade the worker FortiGate, the upgrade will be launched on the SLBC config master FortiGate.

Related Articles

Upgrading Cluster members in SALB(Session Aware Load Balancing) Cluster

Technical Note: Explanation of 'min-links' and 'link-failure-threshold' in HA