Shutdown and startup¶

Use this procedure for a planned shutdown of the whole cloud. Complete the sections in order. Skip components that are not present.

Some deployments use the Ansible control host as the seed hypervisor. In this case, shut it down once, as the final step.

Shutdown¶

Pre-flight¶

Get the MariaDB password on the Ansible control host:

ansible-vault view $KAYOBE_CONFIG_PATH/kolla/passwords.yml \
    --vault-password-file <vault password file> | \
    grep '^database_password:'

ssh stack@<controller>

Run the following checks on every controller.

Galera¶

Enter the MariaDB password when prompted:

docker exec -i mariadb mysql -u root -p \
    -e "SHOW STATUS LIKE 'wsrep_local_state_comment'"

The result must be Synced:

wsrep_local_state_comment  Synced

RabbitMQ¶

docker exec rabbitmq rabbitmqctl cluster_status

Check the output:

Running Nodes lists every controller.
Network Partitions is (none).
Alarms is (none).

Keepalived¶

docker logs keepalived 2>&1 | \
    grep -E 'Entering (MASTER|BACKUP) STATE' | tail -1

One controller must show Entering MASTER STATE. The other controllers must show Entering BACKUP STATE.

Do not continue until all checks pass.

MariaDB backup¶

Run on the Ansible control host:

kayobe overcloud database backup

Check that the command succeeds. See Database Backups for off-host backups.

Instances¶

Workload instances¶

Ask users to stop their instances.

Ironic compute instances¶

Warning

If compute nodes are deployed as Ironic instances, stop all workload instances before stopping the compute node instances.

Stop each Ironic compute instance:

openstack server stop <compute node instance>

Remaining instances¶

Stop any remaining active instances:

for server in $(openstack server list --all-projects --status ACTIVE \
    -f value -c ID); do
    openstack server stop "$server"
done

Wait until this command returns no instances:

openstack server list --all-projects --status ACTIVE

Investigate any failures.

Bifrost¶

On the seed VM, put every managed node into maintenance:

docker exec bifrost_deploy bash -c '
for node in $(baremetal --os-cloud bifrost node list -f value -c UUID); do
    baremetal --os-cloud bifrost node maintenance set \
        --reason full-shutdown "$node"
done'

OpenStack services¶

Run on the Ansible control host:

kayobe overcloud service stop --yes-i-really-really-mean-it

This stops all Kolla containers, including monitoring. The containers start when their hosts boot.

Compute nodes¶

Bifrost-managed compute nodes¶

On the seed VM, shut down each compute node:

docker exec bifrost_deploy baremetal --os-cloud bifrost \
    node power off --soft <compute node>

Wait for each node to reach power off.

Monitoring nodes¶

On the seed VM, shut down each monitoring node:

docker exec bifrost_deploy baremetal --os-cloud bifrost \
    node power off --soft <monitoring node>

Wait for each node to reach power off.

Network nodes¶

On the seed VM, shut down each network node:

docker exec bifrost_deploy baremetal --os-cloud bifrost \
    node power off --soft <network node>

Wait for each node to reach power off.

Controllers¶

On the seed VM, shut down each controller:

docker exec bifrost_deploy baremetal --os-cloud bifrost \
    node power off --soft <controller>

Wait for each node to reach power off.

Ceph¶

Warning

Stop all external Ceph client I/O before continuing. Unmount CephFS, disconnect RBD clients, and stop traffic to RGW/S3 and other Ceph gateways.

On a MON node, check that Ceph reports HEALTH_OK and all placement groups are active+clean, then set noout:

sudo cephadm shell -- ceph -s
sudo cephadm shell -- ceph osd set noout

On the seed VM, shut down Ceph nodes:

docker exec bifrost_deploy baremetal --os-cloud bifrost \
    node power off --soft <Ceph node>

Shut down nodes running MON services last. Wait for each node to reach power off.

Seed VM¶

If the seed is a VM on the seed hypervisor, run:

virsh shutdown <seed VM>

Wait for the VM to stop. If the seed is hosted elsewhere, shut it down using the platform that hosts it.

Seed hypervisor¶

Run on the separate seed hypervisor:

sudo systemctl poweroff

Ansible control host¶

Shut down the Ansible control host last:

sudo systemctl poweroff

Startup¶

Ansible control host¶

Power on the Ansible control host.

Seed hypervisor and seed VM¶

If the seed is a VM on the seed hypervisor, power on the hypervisor if it is separate from the Ansible control host, then start the seed VM:

virsh start <seed VM>

If the seed is hosted elsewhere, start it using the platform that hosts it.

Bifrost¶

On the seed VM, wait for the bifrost_deploy container to be running.

Ceph¶

On the seed VM, start Ceph MON nodes, then the other Ceph nodes:

docker exec bifrost_deploy baremetal --os-cloud bifrost \
    node power on <Ceph node>

On a MON node, wait for all OSDs and placement groups. Continue when Ceph reports HEALTH_OK and all Ceph services are running, then unset noout:

sudo cephadm shell -- ceph orch ls
sudo cephadm shell -- ceph -s
sudo cephadm shell -- ceph osd unset noout

External Ceph clients can now be started.

Controllers and MariaDB¶

On the seed VM, start every controller:

docker exec bifrost_deploy baremetal --os-cloud bifrost \
    node power on <controller>

When all controllers are reachable, recover MariaDB from the Ansible control host:

kayobe overcloud database recover

Network nodes¶

On the seed VM, start every network node:

docker exec bifrost_deploy baremetal --os-cloud bifrost \
    node power on <network node>

Monitoring nodes¶

On the seed VM, start every monitoring node:

docker exec bifrost_deploy baremetal --os-cloud bifrost \
    node power on <monitoring node>

Compute nodes¶

Bifrost-managed compute nodes¶

On the seed VM, start every compute node:

docker exec bifrost_deploy baremetal --os-cloud bifrost \
    node power on <compute node>

Ironic compute instances¶

Warning

If compute nodes are deployed as Ironic instances, start the compute node instances before starting workload instances.

Start the compute node instances:

openstack server start <compute node instance>

Wait for their Nova compute services to report enabled and up:

openstack compute service list --service nova-compute

OpenStack services¶

Repeat the Galera, RabbitMQ and Keepalived checks from Pre-flight.

List containers that are not running:

kayobe overcloud host command run --show-output --command \
    "docker ps --all --filter status=exited --filter status=dead \
    --filter status=restarting --format '{{.Names}}: {{.Status}}'"

No containers should be listed. For a listed container, check its output and the service logs under /var/log/kolla/, then restart its systemd unit:

docker logs <container>
sudo systemctl restart kolla-<container>-container

Bifrost maintenance¶

On the seed VM, remove a healthy node from maintenance:

docker exec bifrost_deploy baremetal --os-cloud bifrost \
    node maintenance unset <node>

Verification¶

Wait for the shutdown alerts to clear in Alertmanager. Investigate any remaining alerts and new errors in OpenSearch Dashboards.

Workload instances¶

Workload instances can now be started.