Running Tempest with Kayobe Automation¶
Overview¶
This document describes how to configure and run Tempest using kayobe-automation from the .automation
submodule included with stackhpc-kayobe-config
.
The best way of running Tempest is to use CI/CD workflows. Before proceeding, consider whether it would be possible to use/set up a CI/CD workflow instead. For more information, see the CI/CD workflows page.
The following guide will assume all commands are run from your
kayobe-config
root and the environment has been configured to run Kayobe
commands unless stated otherwise.
Prerequisites¶
Installing Docker¶
kayobe-automation
runs in a container on the Ansible control host. This
means that Docker must be installed on the Ansible control host if it is not
already.
Warning
Docker can cause networking issues when it is installed. By default, it
will create a bridge and change iptables
rules. These can be disabled
by setting the following in /etc/docker/daemon.json
:
{
"bridge": "none",
"iptables": false
}
The bridge is the most common cause of issues and is usually safe to
disable. Disabling the iptables
rules will break any GitHub actions
runners running on the host.
To install Docker on Ubuntu:
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Installing Docker on Rocky:
sudo dnf install -y dnf-utils
sudo dnf-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Ensure Docker is running & enabled:
sudo systemctl start docker
sudo systemctl enable docker
The Docker buildx
plugin must be installed. If you are using an existing
installation of docker, you may need to install it with:
sudo dnf/apt install docker-buildx-plugin
sudo docker buildx install
# or if that fails:
sudo docker plugin install buildx
Building a Kayobe container¶
Build a Kayobe automation image:
git submodule init
git submodule update
# If running on Ubuntu, the fact cache can confuse Kayobe in the Rocky-based container
mv etc/kayobe/facts{,-old}
sudo DOCKER_BUILDKIT=1 docker build --build-arg BASE_IMAGE=rockylinux:9 --file .automation/docker/kayobe/Dockerfile --tag kayobe:latest .
Configuration¶
Kayobe automation configuration files are stored in the .automation.conf/
directory. It contains:
A script used to export environment variables for meta configuration of Tempest -
.automation.conf/config.sh
.Tempest configuration override files, stored in
.automation.conf/tempest/
and conventionally namedtempest.overrides.conf
ortempest-<environment>.overrides.conf
.Tempest load lists, stored in
.automation.conf/tempest/load-lists
.Tempest skip lists, stored in
.automation.conf/tempest/skip-lists
.
config.sh¶
config.sh
is a mandatory shell script, primarily used to export environment
variables for the meta configuration of Tempest.
See: https://github.com/stackhpc/docker-rally/blob/master/bin/rally-verify-wrapper.sh for a full list of Tempest parameters that can be overridden.
The most common variables to override are:
TEMPEST_CONCURRENCY
- The maximum number of tests to run in parallel at one time. Higher values are faster but increase the risk of timeouts. 1-2 is safest in CI/Tenks/Multinode/AIO etc. 8-32 is typical in production. Default value is 2.KAYOBE_AUTOMATION_TEMPEST_LOADLIST
: the filename of a load list in theload-lists
directory. Default value isdefault
(symlink to refstack).KAYOBE_AUTOMATION_TEMPEST_SKIPLIST
: the filename of a load list in theskip-lists
directory. Default value is unset.TEMPEST_OPENRC
: The contents of anopenrc.sh
file, to be used by Tempest to create resources on the cloud. Default is to read in the contents ofetc/kolla/public-openrc.sh
.
tempest.overrides.conf¶
Tempest uses a configuration file to define which tests are run and how to run
them. A full sample configuration file can be found here. Sensible
defaults exist for all values and in most situations, a blank
*overrides.conf
file will successfully run many tests. It will however also
skip many tests which may otherwise be appropriate to run.
Shakespeare is a tool for generating Tempest configuration files. It contains elements for different cloud features, which can be combined to template out a detailed configuration file. This is the best-practice approach.
Below is an example of a manually generated file including many of the most common overrides. It makes many assumptions about the environment, so make sure you understand all the options before applying them.
[openstack]
# Use a StackHPC-built image without a default password.
img_url=https://github.com/stackhpc/cirros/releases/download/20231206/cirros-d231206-x86_64-disk.img
[auth]
# Expect unlimited quotas for CPU cores and RAM
compute_quotas = cores:-1,ram:-1
[compute]
# Required for migration testing
min_compute_nodes = 2
# Required to test some API features
min_microversion = 2.1
max_microversion = 2.95
# Flavors for creating test servers and server resize. The ``alt`` flavor should be larger.
flavor_ref = <flavor UUID>
flavor_ref_alt = <different flavor UUID>
volume_multiattach = true
[compute-feature-enabled]
# Required for migration testing
resize = true
live_migration = true
block_migration_for_live_migration = false
volume_backed_live_migration = true
[placement]
min_microversion = 1.0
max_microversion = 1.39
[volume]
storage_protocol = ceph
# Required to test some API features
min_microversion = 3.0
max_microversion = 3.70
Tempest configuration override files are stored in
.automation.conf/tempest/
. The default file used is
tempest.overrides.conf
or tempest-<environment>.overrides.conf
depending on whether a Kayobe environment is enabled. This can be changed by
setting KAYOBE_AUTOMATION_TEMPEST_CONF_OVERRIDES
to a different file path.
An overrides.conf
file must be supplied, even if it is blank.
Load Lists¶
Load lists are a newline-separated list of tests to run. They are stored in
.automation.conf/tempest/load-lists/
. The directory contains three objects
by default:
tempest-full
- A complete list of all possible tests.platform.2022.11-test-list.txt
- A reduced list of tests to match the Refstack standard.default
- A symlink toplatform.2022.11-test-list.txt
.
Test lists can be selected by changing KAYOBE_AUTOMATION_TEMPEST_LOADLIST
in config.sh
. The default value is default
, which symlinks to
platform.2022.11-test-list.txt
.
A common use case is to use the failed-tests
list output from a previous
Tempest run as a load list, to retry the failed tests after making changes.
Skip Lists¶
Skip lists are a newline-separated list of tests to Skip. They are stored in
.automation.conf/tempest/skip-lists/
. Each line consists of a pattern to
match against test names, and a string explaining why the test is being
skipped e.g.
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details.*: "Cirros image doesn't have /var/run/udhcpc.eth0.pid"
There is no requirement for a skip list, and none is selected by default. A
skip list can be selected by setting KAYOBE_AUTOMATION_TEMPEST_SKIPLIST
in
config.sh
.
Tempest runner¶
While the Kayobe automation container is always deployed to the ansible control
host, the Tempest container is deployed to the host in the tempest_runner
group, which can be any host in the Kayobe inventory. The group should only
ever contain one host. The seed is usually used as the tempest runner however
it is also common to use the Ansible control host or an infrastructure VM. The
main requirement of the host is that it can reach the OpenStack API.
Tempest CA certificate¶
If your public OpenStack API uses TLS with a Certificate Authority (CA) that is
not trusted by the Python CA trust store, it may be necessary to add a CA
certificate to the trust store in the container that runs Tempest. This can be
done by defining a tempest_cacert
Ansible variable to a path containing the
CA certificate. You may wish to use kayobe_config_path
or
kayobe_env_config_path
to be agnostic to the path where kayobe-config is
mounted within the container. For example:
# Add the Vault CA certificate to the rally container when running tempest.
tempest_cacert: "{{ kayobe_env_config_path }}/kolla/certificates/ca/vault.crt"
Running Tempest¶
Kayobe automation will need to SSH to the Tempest runner (even if they are on
the same host), so requires an SSH key exported as
KAYOBE_AUTOMATION_SSH_PRIVATE_KEY
e.g.
export KAYOBE_AUTOMATION_SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa)
Tempest outputs will be sent to the tempest-artifacts/
directory. Create
one if it does not exist.
mkdir tempest-artifacts
The contents of tempest-artifacts
will be overwritten. Ensure any previous
test results have been copied away.
The Tempest playbook is invoked through the Kayobe container using this
command from the base of the kayobe-config
directory:
sudo -E docker run --name kayobe-automation --detach -it --rm --network host \
-v $(pwd):/stack/kayobe-automation-env/src/kayobe-config -v $(pwd)/tempest-artifacts:/stack/tempest-artifacts \
-e KAYOBE_ENVIRONMENT -e KAYOBE_VAULT_PASSWORD -e KAYOBE_AUTOMATION_SSH_PRIVATE_KEY kayobe:latest \
/stack/kayobe-automation-env/src/kayobe-config/.automation/pipeline/tempest.sh -e ansible_user=stack
By default, no_log
is set to stop credentials from leaking. This can be
disabled by adding -e rally_no_sensitive_log=false
to the end.
To follow the progress of the Kayobe automation container, either remove
--detach
from the above command, or follow the docker logs of the
kayobe
container.
To follow the progress of the Tempest tests themselves, follow the logs of the
tempest
container on the tempest_runner
host.
ssh <tempest-runner>
sudo docker logs -f tempest
Tempest will keep running until completion if the kayobe
container is
stopped. The tempest
container must be stopped manually. Doing so will
however stop test resources (such as networks, images, and VMs) from being
automatically cleaned up. They must instead be manually removed. They should be
clearly labeled with either rally or tempest in the name, often alongside some
randomly generated string.
Outputs¶
Tempest outputs will be sent to the tempest-artifacts/
directory. It
contain the following artifacts:
docker.log
- The logs from thetempest
docker containerfailed-tests
- A simple list of tests that failedrally-junit.xml
- An XML file listing all tests in the test list and their status (skipped/succeeded/failed). Usually not useful.rally-verify-report.html
- An HTML page with all test results including an error trace for failed tests. It is often best toscp
this file back to your local machine to view it. This is the most user-friendly way to view the test results, however can be awkward to host.rally-verify-report.json
- A JSON blob with all test results including an error trace for failed tests. It contains all the same data as the HTML report but without formatting.stderr.log
- The stderr log. Usually not useful.stdout.log
- The stdout log. Usually not useful.tempest-load-list
- The load list that Tempest was invoked with.tempest.log
- Detailed logs from Tempest. Contains more data than theverify
reports, but can be difficult to parse. Useful for tracing specific errors.