Description
Cisco 9800 WLC wireless clients unexpectedly disconnect then reconnect.
It has been observed the 9800 can occasionally return an incomplete list of MAC addresses during an L2 poll. Any online host whose MAC address was not included in the list is marked "offline" in the appliance database.
When this occurs and a subsequent poll returns the full list of online MAC addresses, the host previously marked offline will be re-evaluated for policy assignment.
In some cases, this can lead to a disconnect request being sent to the controller to re-authenticate and re-assign the correct VLAN for the host via RADIUS.
Workaround:
Under the Polling tab of the Cisco 9800 WLC Device Model, increase the L2 polling interval.
Version 8.x: Navigate to Network Devices > Topology
Version 9.x: Navigate to Network > Inventory
Example:
1) Host connects to SSID and 9800 sends RADIUS Access Request to the appliance.
2) Appliance responds with RADIUS Access Accept assigning VLAN A.
3) Scheduled L2 poll of the Cisco 9800 controller completes - MAC address of the host is returned. Host is online and connected to the correct SSID and VLAN.
4) Scheduled L2 poll of the Cisco 9800 controller - MAC address of the host is not returned due to an incomplete list. Host record is marked "offline".
5) Scheduled L2 poll of the Cisco 9800 controller - MAC address of the host is returned. Host record is marked "online".
6) Since a RADIUS request was not received at the time the host was detected back online, a disconnect request is sent to the 9800 to force re-authentication.
7) Host reconnects to SSID and 9800 sends RADIUS Access Request to the appliance.
8) Appliance responds with RADIUS Access Accept assigning VLAN A.
Diagnose.
1) Enable the following debug in the appliance and gather logs to submit to Support. Reference this KB article. See related KB article below for instructions.
nacdebug -name Airespace true
nacdebug -name SnmpV1 true
2) Gather information confirming the 9800 is not returning the full MAC address list. The 9800 returns the MAC address list when responding to SNMP query for macPrefix (1.3.6.1.4.1.14179.2.1.4.1.1) during L2 poll. Snmpwalk can be performed using the appliance CLI or using some other SNMP tool that has access to the device. Type the following in the CLI:
SNMP version 1 or 2.
snmpwalk -v 1 -c <community string> <device ip_address> 1.3.6.1.4.1.14179.2.1.4.1.1
SNMP version 3.
snmpwalk -v 3 -u <username> -l <AuthPriv or NoAuthPriv> -a <MD5 or SHA> -A <password> -x <DES or AES> -X <password> <device ip_address> 1.3.6.1.4.1.14179.2.1.4.1.1
*May need to use single quotes (') with passwords.
Note.
The behavior is random. The several query attempts may be required before an incomplete list is returned.
FortiNAC reads the following SNMP tables when performing an L2 poll of the 9800 (“Airespace” debug prints these values in output.master):
macPrefix = 1.3.6.1.4.1.14179.2.1.4.1.1 <<<<<<< Contains MAC address table listing
ipAddrPrefix = 1.3.6.1.4.1.14179.2.1.4.1.2
apMacPrefix = 1.3.6.1.4.1.14179.2.1.4.1.4
ssidPrefix = 1.3.6.1.4.1.14179.2.1.4.1.7
statusPrefix = 1.3.6.1.4.1.14179.2.1.4.1.9
mobilityStatusPrefix = 1.3.6.1.4.1.14179.2.1.4.1.11
ifNamePrefix = 1.3.6.1.4.1.14179.2.1.4.1.27
ifIdPrefix = 1.3.6.1.4.1.14179.2.1.4.1.29
Lines similar to the following will be seen in the output.master log file when 1st successful poll after host had been marked offline has been completed (step 5).
----------------------------------
yams INFO :: 2021-03-25 13:30:10:576 :: PollThread-poll0 wlc-9800 [80] client mac = 24:ee:9a:b7:56:3d client ip = 144.35.26.112 ifName = staff mobStatus = 1 ap = cc:7f:75:af:36:00 status = 3 <<< MAC address detected via L2 poll
yams.Airespace INFO :: 2021-03-25 13:30:12:825 :: com.bsc.plugin.forwarding.AirespaceApPlugin.checkVlan for client 24:EE:9A:B7:56:3D, element 192.168.80.20 and port 57025
yams.Airespace INFO :: 2021-03-25 13:30:12:826 :: isClientManaged client 24:EE:9A:B7:56:3D 2297954 auth type = 2
yams INFO :: 2021-03-25 13:30:12:827 :: 2021/03/25 13:30:12.827 DevicePluginThread2 client=2297954
yams INFO :: 2021-03-25 13:30:12:827 :: 2021/03/25 13:30:12.827 DevicePluginThread2 clearClient new client= 2297954
yams INFO :: 2021-03-25 13:30:12:827 :: 2021/03/25 13:30:12.827 DevicePluginThread2 evaluateClient: client 2297954 not connected to SSID <<<<< indicates previously offline
yams INFO :: 2021-03-25 13:30:12:828 :: 2021/03/25 13:30:12.828 DevicePluginThread2 evaluateClient client 24:EE:9A:B7:56:3D 2297954 auth type = 2
yams INFO :: 2021-03-25 13:30:12:828 :: 2021/03/25 13:30:12.828 DevicePluginThread2 evaluateClient: client 2297954 not connected to SSID <<<<<
yams INFO :: 2021-03-25 13:30:12:828 :: 2021/03/25 13:30:12.828 DevicePluginThread2 evaluateClient: client 2297954 not connected to WAP <<<<<
----------------------------------
Solution
Addressed in version 9.1.6 and greater. L2 poll is now performed using CLI (instead of SNMP) for Cisco WLC's running firmware version 17 or higher.
Related Articles
Technical Note: Gather logs for debugging and troubleshooting
