Technical Note: High CPU Load on One Pod in NCM Environment

[FortiKoala]

Description

High CPU Load on One Pod in NCM Environment

Solution

Issue:  High CPU load on one pod in a multiple pod NCM environment.  

In a NCM environment with Persistent Agents (PA), there is a property value that indicates when a host is believed to have the PA installed.  This property value is shared among the pods.   

The agent will continue to communicate with a pod until that communication is broken.  At which point, the agent will attempt to communicate with another pod.  Therefore, unless agent traffic is blocked between pod locations, it is possible for an agent to be physically connected to one pod, yet communicating with another.

Example:

• Agent on host is connected and communicating to Pod A.
• Host then moves to Pod B.  
• Online status changes to "offline" on Pod A and "online" to Pod B.
• Agent continues to successfully communicate with Pod A.

If Network Sentry sees a host as online and having the PA installed, it will attempt to establish communication with the agent.  These attempts consume CPU.  Since the agent is configured to communicate with a different pod, the attempt fails and a "Lost Contact with Persistent Agent" event is generated (if event is enabled).  Therefore, if the appliance is attempting to communicate with several agents that are successfully talking to another pod, a large quantity of such events would be generated.  


Solution:    

1. Upgrade Network Sentry to version 8.0.4 or higher. 
2. Ensure Persistent Agent version is at least 4.0.3 or higher.
3. Ensure the agent registry setting (allowedServer list) on the hosts contain all the servers in the NCM environment.  This will allow communication with the appropriate Network Sentry appliance as the host moves between pods.  
4. Enable the Require Online Connected Adapter feature:

a.  In the Administrative UI, navigate to
     System > Settings > Security Management -    Persistent Agent.
b.  Check the Require Connected Adapter box.

The above steps will ensure hosts with agents only communicate with the pod to which they are connected. This will, in turn, prevent the appliance from consuming CPU attempting to communicate with agents that are not going to respond.