Description
When on a 5K unit, one or more slave blades are in 'Waiting' status and are not synchronizing with master blade, the system is actively processing data within the 5K cluster:
To confirm this issue, verify if any of factory slave blades are out of sync with the master blade.
Run the following command to check the blades:
# diag sys confsync showcsum debugzone
Solution
There is no way to force a sync between blades via the CLI, as all is handled via the 'confsyncd' process.
Here are three alternative ways to force a sync between the master and the slave blades (do this during a maintenance window).
1) Kill process 'confsyncd' on the master blade and any other problematic slave blades.
# diag sys top <----- On the problematic blades (to make note of the PID of 'confsyncd' process).
# diag sys kill 11 [PID] <----- On master blade.
# diag sys kill 11 [PID] <----- On worker blades that are in 'waiting status'.
2) Another option is to power cycle the affected slave blade(s) that are stuck in 'Waiting' status.
Use the shelf manager to cut power to a slot remotely if there is no physical access to the device.
Follow the steps below :
- Login to the shelf manager.
- Run clia fru <IPMB_address>’ to check if the blade is still present.
# clia fru 96
IPM Sentry Shelf Manager Command Line Interpreter
96: FRU # 0
Entity: (0xa0, 0x60)
Hot Swap State: M4 (Active), Previous: M4 (Active), Last State Change Cause: Normal State Change (0x0)
Device ID String: "FG5005A"
- Deactivate the specific FRU and the IPM led on the blade will be blue.
# clia deactivate 96 0
IPM Sentry Shelf Manager Command Line Interpreter
Command issued via IPMB, status = 0 (0x0)
Command executed successfully
- Activate back this specific FRU and the blade will reboot.
# clia activate 96 0
IPM Sentry Shelf Manager Command Line Interpreter
Command issued via IPMB, status = 0 (0x0)
Command executed successfully
Note:
These commands will not work for some hot swap states like for example 'M7' (communication Lost).
3) Factory Reset affected blades and are added back in to SLBC.
From the CLI enter the following command:
# execute factory reset
