Technical Note: SLBC worker blade out of sync with Config Master

[cskuan]

Description

This article provides some troubleshooting steps for a SLBC worker blade out of sync with the Config Master.

Solution

A few methods can be used to determine if the worker blade is not in sync with the Config Master.

1)  Issue the command ‘get load-balance status’ at the FortiController.

CH1SL1 # get load-balance status
…………….
     Slot  3: Status:Working   Function:Active
       Link:      Base: Up          Fabric: Up 
       Heartbeat: Management: Good   Data: Good 
       Status Message:"Running"
     Slot  5: Status:Dead   Function:Active
       Link:      Base: Up          Fabric: Up
       Heartbeat: Management: Good   Data: Good
       Status Message:"Waiting for configuration sync."  ------> out of sync

The status of an individual worker blade can be viewed from the FortiController.

If a slot/worker blade with a status where both Base and Fabric are “Up” and Management Heartbeat is “Good”, however “Data” status is “Failed” and Status Message is “Waiting for management heartbeat”.  It means the worker blade is out of sync with the Config Master.

2)  Issue the command ‘diag sys confsync status” at the Config Master or the worker blade itself.

CH1SL3 (global) # diag sys confsync status
ELBC: svcgrp_id=1, slot_id=3
ELBC HB devs:
elbc-ctrl/1: active=1, hb_count=5316220
elbc-ctrl/2: active=0, hb_count=0
ELBC mgmt devs:
elbc-base-ctrl: mgmtip_set=1

zone: self_idx:2, master_idx:2
CH1SL3, Master, uptime=5317356.73, priority=0, slot_id=1:3, idx=2, in_sync=1
CH1SL5, Slave, uptime=5317609.31, priority=1, slot_id=1:5, idx=0, in_sync=0
elbc-base-ctrl: state=3(connected), ip=169.254.1.3, last_hb_time=5317508.42, hb_nr=26586383

From the above output,  all worker blades can be seen.  The worker blade can be identified by the “slot_id” field of the format x:y where x is the chassis number and y is the slot number.

The field “in-sync” identifies if the worker blade is in sync with the Config Master.

0 = out of sync;  1 = in sync.

How to identify which part of the configuration is not in sync and fix it

1)  Issue the command ‘diag sys confsync showcsum” at the Config Master and the work blade that is out of sync.

CH1SL3 (global) # diag sys confsync showcsum
debugzone
global: 8e 31 4c c8 18 2e 3a 0d 3b 5c 17 f7 6d d9 a5 25
root: cb e2 40 90 dc 6a d8 06 c5 0d 07 68 38 5d ee 23
exit: be fa 9d 4a 97 6d 86 6f 8c ce 14 14 51 68 00 2a
elbc-mgmt: 11 05 9c 14 df cb 6b a8 b3 71 a9 03 85 80 aa 62
……………..

checksum
global: 8e 31 4c c8 18 2e 3a 0d 3b 5c 17 f7 6d d9 a5 25
root: cb e2 40 90 dc 6a d8 06 c5 0d 07 68 38 5d ee 23
exit: be fa 9d 4a 97 6d 86 6f 8c ce 14 14 51 68 00 2a
elbc-mgmt: 11 05 9c 14 df cb 6b a8 b3 71 a9 03 85 80 aa 62
………………

Compare the results to determine which item is not in sync.

2)  Instead of comparing the whole config file from each of the worker blade, use the following commands to drill in to find out the exact configuration that is not in sync.

(i.)    If it is “global” checksum mismatch / not in sync

Issue command ‘diag sys confsync showcsum 1’

CH1SL5 (global) # diagnose sys confsync showcsum 1
system.global: f8b31181ae4b93ce5a6e8fbece51d2d1
system.accprofile: 7d79452c78377be2616149264a18fd5c
system.npu: 00000000000000000000000000000000
system.vdom-link: 00000000000000000000000000000000
wireless-controller.global: 00000000000000000000000000000000
wireless-controller.vap: 00000000000000000000000000000000
system.switch-interface: 00000000000000000000000000000000
system.lte-modem: 00000000000000000000000000000000

If either one of the items above is mismatched, drill further in by using the same commands followed by the item name, for example:

'diag sys confsync showsum system.switch-interface’

(ii.)  If it is any vdom other than “global”.

Issue the command ‘diag sys confsync showcsum ’ and further drill in by using the same command followed by the item name.

3.    Determine which setting blocked the confsysc process.

4.    Manually change the setting to correct the out of sync problem.

5.    It may be necessary to change the setting on the Config Master unit.