Technical Note: CSTN 00016 - Support for externalizing your CyOPs databases on Amazon

[Andy_G]

Description

CyOPs™ supports externalization of your CyOPs™ databases, i.e., the PostgreSQL database on Amazon RDS, and the MongoDB and ElasticSearch data on Amazon CentOS. Externalization is migration of data from your local database instance to a remote database instance that has same version of Postgresql, MongoDB, or ElasticSearch outside of the CyOPs™ virtual appliance. 

The process of externalizing your Postgresql and MongoDB databases, and ElasticSearch data is explained in the CyOPs™ documentation: Externalization of your CyOPs™ Postgresql and MongoDB databases chapter in the "Administration Guide." The process of externalizing your ElasticSearch data is explained in the CyOPs™ documentation: Elasticsearch Configuration chapter in the "Administration Guide."

Storage

Minimum Recommended Storage Size for the externalized database is 50GB.

Solution

Prerequisites 

• Ensure that your Amazon RDS and your CyOPs™ virtual appliance are in the same region.
• Open port on your Amazon RDS instance using which you will communicate with your CyOPs™ instance. By default, this port is set as 5432.

Creating your PostgreSQL database on Amazon RDS

The process of externalizing your Postgresql is explained in the CyOPs™ documentation: Externalization of your CyOPs™ Postgresql and MongoDB databases chapter in the "Administration Guide."

1. Log into you Amazon account.
2. Navigate to Services > Databases > RDS.
3. Click Create Database as select PostgreSQL as the type of engine.
4. On the Specify DB details page, enter the following details:
   1. In the Instance specifications section, from the DB engine version drop-down list, select PostgreSQL 10.3-R1.
   2. In the Settings section, in the Master username field enter postgres and in the Master password field, enter the password that you want to specify for your PostgreSQL database.
      
   3. Enter all the required details on the Specify DB details page, and click Next.
5. On the Configure Advanced Settings page enter the following details:
   1. In the Database Options section, in the Database name field enter postgres.  
      Important: You must specify the Database name as postgres.
      
   2. (Optional) You can also specify the port using which you will communicate with your CyOPs™ instance.
   3. Enter all the required details on the Configure Advanced Settings page, and click Save to create your PostgreSQL database on Amazon RDS.
6. Check that the connectivity between your Amazon PostgreSQL RDS and your CyOPs™ instance using the user credentials, i.e., username and password, that you have specified in your RDS.
   You can check the connectivity using the following command:
   psql -h <pg_hostname> -U <pg_username> -p <port_no> -l postgres
7. Connect to your postgres database and perform the following steps:
   1. Create the cyberpgsql user using following command:  
      # CREATE USER  cyberpgsql WITH PASSWORD ‘<your password>’  CREATEROLE CREATEDB;
   2. Add the postgres user to the cyberpgsql role using the following command:
      GRANT cyberpgsql TO postgres;
8. To check the connectivity between CyOPs™ and the external RDS, use the following commands:  
   psql -h <external_rds_ip/host> -U cyberpgsql -l postgres

Creating your MongoDB database on Amazon CentOS

The process of externalizing your MongoDB databases is explained in the CyOPs™ documentation: Externalization of your CyOPs™ Postgresql and MongoDB databases chapter in the "Administration Guide."

Prerequisites

You must have version 3.4 of MongoDB installed.

1. We recommend that you setup your MongDB with SSL/TLS certificates. See https://docs.mongodb.com/manual/tutorial/configure-ssl/ for more information. Following are the steps in brief that you can use to setup your MongoDB:
   1. Create the ssl directory in the /var/lib/mongo/ folder and run the following command to change the permission and ownership:
      $ sudo chown -R mongod:mongod /var/lib/mongo/ssl
$ sudo chmod 700 /var/lib/mongo/ssl
   2. You require a .pem file that contains a public key certificate and its associated private key. Copy this .pem file to the /var/lib/mongo/ssl directory, and ensure that the file's ownership is set as mongod:mongod.
   3. To access the mongodb instance from the CyOPs™ instance, open the /etc/mongod.conf file and edit the following:
      1. Change the bindip in the net section to the CyOPs™ IP address from where you are externalizing MongoDB. For more information, see https://docs.mongodb.com/manual/reference/configuration-options/#net-options
      2. To enable SSL, in the net: ssl: section ensure that the location of the server pem file (mentioned in step 2) is correct. For example, /var/lib/mongo/ssl/<file_name>.pem. For more information, see the ssl section in https://docs.mongodb.com/manual/reference/configuration-options/#net-options
   4. Restart the mongodb service, use the following command:
      $ sudo systemctl restart mongod
2. To create the MongoDB database:
   1. Login to MongDB and run the following command:
      use venom
   2. To create the MongoDB user, run the following command:
      db.createUser
( 
   {
       user: “cybermongo”,
       pwd: “<yourMongoDBPassword>”,
       roles: [
                    { role: “readWrite”, db: “venom” }
             ]
     }
)
   3. To check connectivity between CyOPs™ and MongoDB, use the following commands:
      usr/bin/mongo 
--sslAllowInvalidHostnames <MONGO_HOST_EXTERNAL>:3128/venom
--sslPEMKeyFile <path_of_ssl_file> --ssl -u <user_name> -p <_mongo_external_password> --eval "db.stats(1024)"

Creating your ElasticSearch data on Amazon CentOS

The process of externalizing your ElasticSearch data is explained in the CyOPs™ documentation: Elasticsearch Configuration chapter in the "Administration Guide." 

1. In the elasticsearch.yml file, which is located at /etc/elasticsearch/elasticsearch.yml replace network.host: with the IP address from where you are going to receive the input for ElasticSearch. 
2. On the ElasticSearch instance, from the firewall, open port 9200 for the CyOPs™ instance.
3. In the db_config.yml of your CyOPs™ instance located at /opt/cyops/configs/database/db_config.yml, replace localhost with the IP address of the ElasticSearch instance, and update your elasticsearch username and password in the 'es_user' and 'secret' keys, respectively.
   This should index all the ElasticSearch data from the ElasticSearch instance to the ElasticSearch of the first instance (mentioned in step 1).
4. (Optional) If you are externalizing an ElasticSearch instance that already contains some data, then you can run the following sync command to do an initial indexing of the CyOPs™ data into ElasticSearch:
   sudo -u nginx php /opt/cyops-api/app/console cybersponse:elastic:create --env="prod"