ci-builder

The ci-builder Kayobe environment is used to build Kolla container images. Images are built using package repositories in the StackHPC Ark Pulp service, and pushed there once built.

Warning

This guide was written for the Yoga release and has not been validated for Master. Proceed with caution.

In general it is preferable to use the container image build CI workflow to build container images, but this manual approach may be useful in some cases.

See also

A container image build GitHub Actions workflow using this environment is described here.

See also

An overcloud host image build GitHub Actions workflow using this environment is described here.

Prerequisites

  • a Rocky Linux 9 or Ubuntu Noble 24.04 host

Setup

Access the host via SSH. You may wish to start a tmux session.

If using an LVM-based image, extend the lv_home and lv_tmp logical volumes.

sudo pvresize $(sudo pvs --noheadings | head -n 1 | awk '{print $1}')
sudo lvextend -L 4G /dev/rootvg/lv_home -r
sudo lvextend -L 4G /dev/rootvg/lv_tmp -r

Install package dependencies.

On Rocky Linux:

sudo dnf install -y git

On Ubuntu:

sudo apt update
sudo apt install -y gcc git libffi-dev python3-dev python-is-python3 python3-venv

Clone the Kayobe and Kayobe configuration repositories (this one):

cd
mkdir -p src
pushd src
git clone https://github.com/stackhpc/stackhpc-kayobe-config -b stackhpc/2025.1 kayobe-config
popd

Create a virtual environment and install Kayobe:

cd
mkdir -p venvs
pushd venvs
python3 -m venv kayobe
source kayobe/bin/activate
pip install -U pip
pip install -r ../src/kayobe-config/requirements.txt
popd

Add initial network configuration:

sudo ip l add breth1 type bridge
sudo ip l set breth1 up
sudo ip a add 192.168.33.3/24 dev breth1
sudo ip l add dummy1 type dummy
sudo ip l set dummy1 up
sudo ip l set dummy1 master breth1

On Ubuntu systems, persist the running network configuration.

sudo cp /run/systemd/network/* /etc/systemd/network

Installation

Acquire the Ansible Vault password for this repository, and store a copy at ~/vault-pw.

The following commands install Kayobe and its dependencies, and prepare the Ansible control host.

export KAYOBE_VAULT_PASSWORD=$(cat ~/vault-pw)
pushd ~/venvs/kayobe
source bin/activate
popd
pushd ~/src/kayobe-config
source kayobe-env --environment ci-builder
kayobe control host bootstrap

Deployment

If using an LVM-based image, uncomment the seed_lvm_groups variable in etc/kayobe/environments/ci-builder/seed.yml.

If using an LVM-based image, grow the root volume group.

kayobe playbook run etc/kayobe/ansible/tools/growroot.yml -e growroot_group=seed

On Ubuntu systems, purge the command-not-found package.

kayobe playbook run etc/kayobe/ansible/fixes/purge-command-not-found.yml

Next, configure the host OS & services.

kayobe seed host configure

Authenticating Pulp proxy

If you are building against authenticated package repositories such as those in Ark, you will need to provide secure access to the repositories without leaking credentials into the built images or their metadata. This is typically not the case for a client-local Pulp, which provides unauthenticated read-only access to the repositories on a trusted network.

Docker provides build secrets, but these must be explicitly requested for each RUN statement, making them challenging to use in Kolla.

StackHPC Kayobe Configuration provides support for deploying an authenticating Pulp proxy that injects an HTTP basic auth header into requests that it proxies. Because this proxy bypasses Pulp’s authentication, it must not be exposed to any untrusted environment.

Ensure that localhost is resolvable if Docker bridge networking is disabled. This may be achieved by adding the following to /etc/hosts:

127.0.0.1 localhost

To deploy the proxy:

kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp/pulp-auth-proxy.yml

Building images

At this point you are ready to build and push some container images.

kayobe seed container image build --push
kayobe overcloud container image build --push

If using an authenticating Pulp proxy, append -e stackhpc_repo_mirror_auth_proxy_enabled=true to these commands.

The container images are tagged as 2025.1-<distribution>-<datetime>. Check tag in /opt/kayobe/etc/kolla/kolla-build.conf or run docker image ls to see the tag of the new images.

To build images for a different base distribution, set -e kolla_base_distro=<distro>.

To build images using a specific tag, set -e kolla_tag=<tag>.

Using the new images

To use the new images, edit ~/src/kayobe-config/etc/kayobe/kolla-image-tags.yml to reference the tag.