Technical Note: CSTN 00012 - Troubleshooting CyOPs Configuration Wizard

[Andy_G]

Description

Applicable to: CyOPs™ 4.11.1

When the CyOPs™ Configuration Wizard is being run, and it displays a message, such as:

One or more tasks failed. Please check the configuration log for details.

You can troubleshoot the CyOPs™ Configuration Wizard errors by checking its logs that are located at /var/log/cyops/config-vm-<timestamp>.log

The CyOPs™ Configuration Wizard log file might contain any one of the following errors, resolutions for which are mentioned in this article.

Solution

1. RabbitMQ service error

Description

CyOPs™ Configuration Wizard log contains the following error:

Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
    fs.file-max = 100000
    fs.file-max = 100000
    User process started.
    Error: unable to perform an operation on node 'rabbit@tester'. Please see diagnostics information and suggestions below.

Root cause

The most common reasons for the RabbitMQ service errors are:

• Target node is unreachable (e.g., due to hostname resolution, TCP connection or firewall issues) 
• CLI tool fails to authenticate with the server (e.g., due to CLI tool's Erlang cookie not matching that of the server) 
• Target node is not running

To further diagnose the issue, do the following: 

• See the CLI, and to learn more see the RabbitMQ clustering and networking guides on http://rabbitmq.com/documentation.html
• Consult the server logs on for the node rabbit@tester

Resolution

RabbitMQ service errors occur when the hostname change script does not work as expected.

To resolve this issue, do the following:

1. Run the following command:
   sh /opt/cyops/configs/scripts/regeneratedefaultconfigs.sh -h <hostname>
   where <hostname> is the hostname for your CyOPs™ instance, as reported by the hostname command.
2. Re-run the CyOPs™ Configuration Wizard.

2. Mongodb service error

Description

• Check the mongodb log located at /var/log/mongodb/mongod.log, it might contains the an error such as:
  2018-08-17T05:43:11.061+0000 I CONTROL [signalProcessingThread] shutting down with code:0 2018-08-17T05:43:15.247+0000 I CONTROL [main] ***** SERVER RESTARTED ***** 2018-08-17T05:43:15.251+0000 E NETWORK [main] cannot read certificate file: /var/lib/mongo/ssl/server.leaf.pem1 error:02001002:system library:fopen:No such file or directory 2018-08-17T05:43:15.255+0000 F CONTROL [main] Failed global initialization: InvalidSSLConfiguration: Can not set up PEM key file.
• Check the mongdb service using the following command:
  systemctl status mongodb
  The mongodb service might display an error such as:
  [root@bb scripts]# systemctl status mongod mongod.service - High-performance, schema-free document-oriented database Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2018-08-17 05:43:15 UTC; 4s ago Docs: https://docs.mongodb.org/manual Process: 12781 ExecStart=/usr/bin/mongod $OPTIONS (code=exited, status=1/FAILURE) Process: 12778 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS) Process: 12776 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS) Process: 12773 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS) Main PID: 19994 (code=exited, status=0/SUCCESS) Aug 17 05:43:15 bb.cybersponse.net systemd[1]: Starting High-performance, schema-free document-oriented database... Aug 17 05:43:15 bb.cybersponse.net mongod[12781]: about to fork child process, waiting until server is ready for connections. Aug 17 05:43:15 bb.cybersponse.net mongod[12781]: forked process: 12784 Aug 17 05:43:15 bb.cybersponse.net mongod[12781]: ERROR: child process failed, exited with error number 1 Aug 17 05:43:15 bb.cybersponse.net systemd[1]: mongod.service: control process exited, code=exited status=1 Aug 17 05:43:15 bb.cybersponse.net systemd[1]: Failed to start High-performance, schema-free document-oriented database. Aug 17 05:43:15 bb.cybersponse.net systemd[1]: Unit mongod.service entered failed state. Aug 17 05:43:15 bb.cybersponse.net systemd[1]: mongod.service failed.

Root cause

• Invalid SSL certificates
• Mongodb service error

Resolution

In case the error is due to invalid SSL certificates, ensure that a valid certificate is present at /var/lib/mongo/ssl/server.leaf.pem.

If the error is due to an invalid SSL certificate, then do the following:

1. Regenerate SSL certificates using the following command:
   sh /opt/cyops/configs/scripts/regeneratedefaultconfigs.sh -c <hostname>
2. Start the mongodb service.

In case of a mongodb service error, take the appropriate steps based on the error mentioned in the log file.

3. Appliance Key error

Description

Post-VM configuration, while executing playbooks you might observe the following error:

Unable to load API credentials from cache or DAS.’, ‘hydra:title’: ‘An error occurred’}

Root cause

Invalid Appliance Key.

Resolution

1. Rerun the following command:
   /opt/cyops/configs/scripts/regenerate_default_configs.sh -k
2. Restart all CyOPs™ services using the following command:
   /opt/cyops/configs/scripts/services.sh restart