Technical Tip: CSTN 00043 - CyOPs Performance Benchmarking for v5.0.1

[Andy_G]

Description
This article describes the performance benchmark tests conducted in CyberSponse labs.
The performance benchmarking tests were performed on CyOPs™ version 5.0.1 Build 098.

Solution
Objective.

The objective of this performance test is to measure the time taken to create alerts in CyOPs™, and complete the execution of corresponding playbooks on the created alerts on a single-node CyOPs™ appliance and a cluster setup of CyOPs™.

The data from this benchmark test can help you in determining your scaling requirements for CyOPs™ instance to handle the expected workload in your environment.

Environmnent.

CyOPs™ Virtual Appliance Specifications

Component	Specifications
CPU	8 CPUs
Memory	16 GB
Storage	250 GB virtual disk running on top of Samsung SSD 360 Pro model attached to VMware ESX server

Operating System Specifications

Operating System	Kernel Version
CentOS 7	3.10.0-957.5.1.eI7.x86_6

Pre-test Conditions

At the start of each test run -

• The test environment contained zero alerts.
• The test environment contained only the CyOPs™ built-in connectors such as IMAP, Utilities, etc.
• The system playbooks were deactivated and there were no running playbooks.
• The playbook execution logs were purged.

Details of the CyOPs™ Performance Benchmarking Test

The test was executed using an automated testbed that initiated HTTPS calls per clock tick (X alert API calls triggered at once) which created alerts in CyOPs™ and then triggered a playbook for each alert created. Steps are as follows:

1. The alerts were created using JMeter to simulate parallel invocation of the API - ‘/api/3/alerts/’.
2. When an alert was created, a post-create playbook is triggered which performs the following steps: 
   • Declare variables using the Set Variable Step
   • Extract artifacts from the source data of the alert using the “CyOPs: Extract Artifacts from String” action of the CyOPs Utilities connector.
   • Add the extracted artifacts in the “Closure Notes” field of the alert.
   • Update the status of the alert to “Closed”.

Setup

The test is invoked in two different setups:

•  A single node CyOPs™ machine : This machine is standalone machine for CyOPs™ and all calls are directly ingested by this machine.
• A High Availability (HA) two-node cluster with an HA Proxy: This setup is a cluster of two CyOPs™ machines which are joined in the Active-Active state using the CyOPs™ HA feature. This setup also contains an HA proxy, which acts as a load balancer. All the calls are directed to the HA proxy load balancer, which functions in a round-robin method to distribute the load of calls.

Observations

The data in the following tables outlines the number of alerts ingested in a clock tick, the total time taken to ingest those alerts, and the total time taken for all the playbooks triggered to finish execution.

Single Invocation Test run on a single-node CyOPs™ appliance 

Number of alerts created in CyOPs™	Total time taken to create all alerts in CyOPs™ (in seconds)	Total time taken to execute all Playbooks (in seconds)
25	2	10.603
50	4	22.578
100	9	45.108

Single Invocation Test run on a two-node Active-Active CyOPs™ cluster 

Number of alerts created in CyOPs™	Total time taken to create all alerts in CyOPs™ (in seconds)	Total time taken to execute all Playbooks (in seconds)
25	3	4.670
50	3	5.704
100	7	14.616

Sustained Invocation Test.

In the sustenance test conducted on a two-node Active-Active CyOPs™ cluster, we could ingest 200 Alerts every 20 secs over 12 hours and observed that 202788 alerts were generated and corresponding playbooks successfully completed.

In the sustenance test conducted on a single node machine we could ingest 100 Alerts every 20 secs over 12 hours and observed that 117592 alerts were generated and corresponding playbooks successfully completed.

* The number of alerts ingested in the system are the same as the alerts generated by the performance tool.

Conclusion.

Based on this test, we conclude that CyOPs™ could process an average of 9799 alerts in an hour in a single node and 16788 alerts in an hour in a two-node Active-Active CyOPs™ cluster. This includes creation of alerts, and running corresponding playbooks to process the alerts.

Notes.

In a production environment the following factors might vary, which could affect the observations:

1) The size of alert data.

2) The number of playbooks that are being executed in parallel for each alert (ex, system playbook for notification or triage/investigate playbooks).

3) The number of steps in each playbook.

4) The network bandwidth especially for outbound connections to applications such as VirusTotal.