2026.1 Gazpacho Series Release Notes

stackhpc/18.4.0.32

New Features

  • Uses the StackHPC Ironic fork to include diskless boot support and architecture-specific virtual media ESP bootloader selection.

stackhpc/18.4.0.30

New Features

  • Bumps Ceph to Squid v19.2.4.

stackhpc/18.4.0.28

New Features

  • Adds Cinder support for VAST storage backends and Manila support for VAST multitenancy.

stackhpc/18.4.0.27

Security Issues

  • Fixes CVE-2026-42998, CVE-2026-42999, CVE-2026-43000, CVE-2026-43001 and CVE-2026-44394 with updated Keystone images.

stackhpc/18.4.0.26

New Features

  • Adds Rocky Linux aarch64 all-in-one CI jobs.

stackhpc/18.4.0.24

New Features

  • The smartmon exporter now treats historical SMART temperature and airflow threshold breaches as non-critical when calculating the smartmon_device_smart_healthy metric. This prevents disks with only a past over-temperature event from being reported as actively unhealthy.

    A new smartmon_device_historical_temperature_failure metric is exported so these historical temperature or airflow threshold breaches can still be viewed and alerted on separately as a warning if required.

stackhpc/18.4.0.21

Bug Fixes

  • Support use of pulp_enable_tls in combination with the newer pulp collection by configuring requests to use CA bundle where internal PKI is configured.

stackhpc/18.4.0.20

New Features

  • Updates to IPA image selection to continue using the CentOS Stream 9 images with Rocky 10, until Rocky 10 IPA images are stable.

  • IPA building now uses the Kayobe ipa_build_distro and ipa_build_release overrides to build, in preparation for building pure Rocky IPA images.

stackhpc/18.4.0.18

Bug Fixes

  • Fix for pulp artifact promotion playbook following on from updates to pulp.squeezer collection - the base_path attribute cannot be used when querying a file distribution.

stackhpc/18.4.0.14

Bug Fixes

  • Updates Nova container images to fix LP#2140347, where persistent disk detach could incorrectly log a failure warning.

stackhpc/18.4.0.13

New Features

  • Adds support for the Rocky Linux security repository. This repository is disabled by default, like in Rocky Linux. It can be enabled by setting dnf_enable_rocky_security to true.

stackhpc/18.4.0.11

New Features

  • Container images for Epoxy can now be built on Rocky 10.

stackhpc/18.4.0.10

New Features

  • Added four playbooks for migrating secret store from Hashicorp Vault to OpenBao.

    vault-bao-migration-seed.yml does Vault to OpenBao migration on seed node. This is single node migration, so API calls to seed secret-store can be disruptive for short period of time.

    vault-bao-migration-overcloud.yml does Vault to OpenBao migration on overcloud. (Default: controller nodes) This is HA migration, so no downtime is expected.

    vault-bao-migration-change-config.yml automatically update SKC to target OpenBao. It is recommended to use this playbook after all Vault deployments are migrated to OpenBao.

    vault-bao-migration-all.yml runs all other three playbooks in order.

    As Hashicorp Vault will no longer be supported from OpenStack 2026.1, it is strongly recommended to migrate to OpenBao before upgrading.

stackhpc/18.4.0.9

New Features

  • The pulp.squeezer collection has been updated to the latest version (0.3.0), with dependencies added and updated as required. The stackhpc.pulp collection has also been updated to it’s latest version (0.6.0). Combined, these two updates drastically improve the time taken for artifacts to be pushed to and pulled from Ark.

Upgrade Notes

  • New versions of pulp.squeezer (0.3.0) and stackhpc.pulp (0.6.0) are available, these collections and their dependencies can be updated by running a control host bootstrap.

stackhpc/18.4.0.7

Bug Fixes

  • DNF repository URLs now use $basearch instead of kolla_base_arch so hosts and build containers resolve the repository architecture locally.

stackhpc/18.4.0.3

Security Issues

  • Bumps repository versions to includes fixes for CVE-2026-31431 (Copy Fail).

stackhpc/18.4.0.2

New Features

  • Adds support for syncing x86_64 and aarch64 RPM repositories from StackHPC Ark into local Pulp using stackhpc_pulp_rpm_architectures.

stackhpc/18.4.0.1

New Features

  • Added support for DOCA OFED on Rocky Linux 9 aarch64.

stackhpc/18.1.0.266

New Features

  • Adds a stackhpc_enable_ceph_cow_optimisations feature flag to enable copy on write optimisations when using Ceph. Please see the Copy on write optimisations section under Configuraton Guide > Cephadm and Kayobe in the documentation.

    The feature is currently opt-in. Note the documented permissions for the images pool for the Cinder user have been adjusted to make this easier to apply for future deployments.

stackhpc/18.1.0.265

New Features

  • Update squid-proxy seed image to one that provides Squid 6.10. This version Squid does not support the dns_v4_first configuration directive, so operators using walled-garden networking should ensure that DNS AAAA records returned from the DNS server correspond to routable IPv6 addresses.

stackhpc/18.1.0.264

New Features

  • CIS benchmark hardening has been implemented for Rocky 10 using the RHEL10-CIS role. Rules overridden have been mapped to the overrides previously used for Rocky 9.

stackhpc/18.1.0.260

Bug Fixes

  • Fixes an issue where Ironic would not be enabled after including baremetal mixin in the configuration.

stackhpc/18.1.0.259

Upgrade Notes

  • In ML2/OVS deployments, Neutron security group rules will now be installed in nftables to align with the behaviour from the 2024.1 release. If you are running a 2025.1 release older than this one, please run the following commands after upgrading the Neutron containers to avoid conflicts between iptables-legacy and iptables-nft rules (this operation will cause downtime):

    kayobe playbook run \
  $KAYOBE_CONFIG_PATH/ansible/fixes/flush-iptables-legacy.yml \
  $KAYOBE_CONFIG_PATH/ansible/fixes/rabbitmq-reset.yml

    You can check if Neutron has installed legacy iptables rules by running:

    iptables-save-legacy | grep neutron

    If you are upgrading directly to this release or a newer one, no action is required.

Bug Fixes

  • Fixed an issue where Neutron security group rules were being created as legacy iptables rules instead of nftables rules. The expected behaviour is that these rules are created using the iptables-nft compatibility package, matching the behaviour introduced in the 2024.1 release.

stackhpc/18.1.0.258

New Features

  • Add Rocky Linux 9 and Rocky Linux 10 aarch64 overcloud host image build and promotion support. The architecture consumed from Ark can be selected with stackhpc_overcloud_host_image_arch.

stackhpc/18.1.0.257

New Features

  • Add support for building multipathd Kolla container images.

stackhpc/18.1.0.256

New Features

  • Added support for building Rocky Linux 9 Ironic Python Agent (IPA) images for the aarch64 architecture in the IPA image CI workflow. The resulting images are published under a separate aarch64 Ark path and can be selected in configuration with stackhpc_ipa_arch. Rocky Linux 9 now also has a separate stackhpc_rocky_9_ipa_image_version_aarch64 version pin for the aarch64 IPA image.

stackhpc/18.1.0.251

Bug Fixes

  • Updates Magnum to a newer Epoxy release version following critical bugs merged & backported. Also, updates Helm version to latest (4.1.4).

stackhpc/18.1.0.244

Security Issues

stackhpc/18.1.0.243

Bug Fixes

  • Updates Neutron images to fix support for trunks on bond ports when using Dell OS10 switches.

stackhpc/18.1.0.242

Bug Fixes

  • Reduce the maximum number of PULP_CONTENT_WORKERS and PULP_API_WORKERS to 12 to prevent connection starvation on seed hosts with more than 12 cpu threads.

stackhpc/18.1.0.238

New Features

  • Added the baremetal mixin environment. This is an opt-in feature. Please see the docs for more details.

stackhpc/18.1.0.234

New Features

  • The Octavia Amphora image was rebuilt and now uses Ubuntu 24.04 LTS instead of Ubuntu 22.04 LTS.

stackhpc/18.1.0.232

New Features

  • Host images for Rocky Linux 10 can now be built in the 2025.1 release

Upgrade Notes

  • Updates stackhpc-image-elements to v1.6.6

stackhpc/18.1.0.229

Bug Fixes

  • Ensure that seed hosts using Podman pull seed container images using their fully-qualified name, rather than their short-name.

  • Ensure that the container_engine variable is respected when running pre.yaml and post.yaml for seed containers.

  • Replace instances of kolla_container_engine with container_engine.

stackhpc/18.1.0.223

New Features

  • Added Rocky Linux 10 repositories for 2025.1 release

stackhpc/18.1.0.221

Bug Fixes

  • Fixed an issue with the get-nvme-drives playbook where it would fail if no NVMe drives were present.

stackhpc/18.1.0.215

Bug Fixes

stackhpc/18.1.0.214

Bug Fixes

  • Fixes CI multiarch manifest creation by setting pipefail in the manifest script. Failures from docker manifest create or docker manifest push now fail the job even when piped to tee.

stackhpc/18.1.0.212

New Features

  • New logic for installing grafana plugins. When building you can pass a variable grafana_plugins_append that contains a list of plugins you want to install. This will be appended to the default list of plugins. The existing method of overwriting the grafana_plugins_install block still works.

stackhpc/18.1.0.208

Bug Fixes

  • Fixes an issue where the Octavia health worker gets stuck in an infinite loop if the connection to the database is interrupted. See LP#2129562 for more details. Also pulls in other fixes from the upstream stable/2025.1 branch for Octavia. See: https://github.com/stackhpc/octavia/pull/59

stackhpc/18.1.0.207

Bug Fixes

  • Updates Nova container images on Rocky Linux 9 to resolve the following issue: operation failed: guest CPU doesn't match specification: missing features: pdcm.

stackhpc/18.1.0.203

Bug Fixes

  • Fixes an issue with Ironic rebuilds when the client was using Nova Compute API version >= 2.93. This caused servers to enter into the error state when a rebuild was requested. See LP#2127017 for more details.

stackhpc/18.1.0.199

Security Issues

stackhpc/18.1.0.188

New Features

  • Adds a basic NGS driver for Dell SONiC switches.

stackhpc/18.1.0.187

Security Issues

  • Rebuilds all container images to address OpenSSL vulnerabilities, including CVE-2025-15467.

stackhpc/18.1.0.186

New Features

  • Adds an extra delay after reboot in the maintenance/reboot.yml playbook to ensure hosts can be reached reliably. The default delay is 5 seconds and can be customised with the post_reboot_delay_s variable.

stackhpc/18.1.0.184

Security Issues

  • Bumps Rocky Linux 9.7 packages to address CVE-2025-68285 in Ceph client session initialization in the Linux kernel.

stackhpc/18.1.0.183

New Features

  • Adds a definition for NVIDIA H200 NVL to the stackhpc_gpu_data.

stackhpc/18.1.0.181

Bug Fixes

  • Rebuilds nova and neutron images to bring in upstream fixes to networking-mlnx agent. Please see stackhpc/networking-mlnx#9 for more details.

stackhpc/18.1.0.172

New Features

  • os-capacity v0.7 provides aarch64 container builds.

Bug Fixes

  • os-capacity v0.7 includes memory leak fixes.

stackhpc/18.1.0.171

New Features

  • Added support for Rocky Linux 9.7, including host packages and a full container image refresh.

  • 9.7 is now the default release for Rocky Linux.

Upgrade Notes

  • The upgrade to DOCA 3.2.1 is required for Rocky Linux 9.7.

stackhpc/18.1.0.167

New Features

  • Adds networking-generic-switch support for bond interfaces in trunk mode on Arista switches.

stackhpc/18.1.0.163

Bug Fixes

  • Fix missing NTP configuration for infrastructure VMs. Operators should run kayobe infra vm host configure --tags ntp.

stackhpc/18.1.0.158

Bug Fixes

  • Fixes an issue with blackbox exporter endpoint generation when prometheus server and blackbox exporter are not colocated.

stackhpc/18.1.0.156

Bug Fixes

  • The RHEL9 CIS Rule 1.2.2 has been disabled, to stop globally enabling gpgchecks on all yum repos. This fixes an issue where custom repos would have the check incorrectly enabled, even when explicitly configured not to.

stackhpc/18.1.0.146

Bug Fixes

  • The default Rocky 9 overcloud host image has been updated to include upstream package repos out of the box

stackhpc/18.1.0.141

New Features

  • Add playbooks and configuration to enable the easy deployment of Pulp with TLS support in combination with certificates generated via OpenBao.

stackhpc/18.1.0.135

New Features

  • Adds a mixin environment that includes policy overrides to enable a baremetaluser role, that is able to create servers on specific baremetal nodes, with specific IP addresses on a shared network.

stackhpc/18.1.0.124

Bug Fixes

  • CIS hardening playbook skips service accounts that do not exist on the host (e.g. kolla on non-Kolla/Ceph-only nodes) to avoid errors.

stackhpc/18.1.0.123

New Features

  • Updates the stackhpc.cephadm Ansible collection to version 1.22.0, pulling in Tentacle support and recent fixes.

stackhpc/18.1.0.111

New Features

  • Changed the IPA (Ironic Python Agent) image compression algorithm from the default gzip to zstd. This improves provisioning performance by reducing the size of the IPA boot ISO transferred from the Ironic conductor to the baremetal nodes.

stackhpc/18.1.0.103

New Features

  • Added a playbook for unsealing secret store deployed at CI runners (Hosts in github-runners or gitlab-runners inventory group). To run CI after rebooting CI runners, secret store needs to be unsealed using this playbook.

stackhpc/18.1.0.102

Bug Fixes

  • Fixes Rocky Linux host image builds not using repositories from Ark.

stackhpc/18.1.0.101

Bug Fixes

  • Updated the OVN chassis priority fix playbook to detect the northbound database leader via ovs-appctl cluster/status, ensuring only the true leader runs the priority alignment.

stackhpc/18.1.0.99

Security Issues

  • Security fixes for bug 2119646: Unauthenticated access to EC2/S3 token endpoints can grant Keystone authorization.

stackhpc/18.1.0.95

Bug Fixes

  • Rebuilt Magnum containers to include fix for clusters failing to create when referencing template by name.

    See bug #2121213 for more details.

stackhpc/18.1.0.89

New Features

  • Add support for building iscsid Kolla container images.

stackhpc/18.1.0.88

New Features

  • Added a new alert RadosGWUsageExporterNotServingMetrics, which will fire when the RadosGW Usage Exporter fails to serve RGW metrics for more than 5 minutes.

stackhpc/18.1.0.81

Bug Fixes

  • Fixes a regression where the checksums files for artifacts uploaded to Ark would not include the filename. This was causing Bifrost to fail when validating the checksums for overcloud host images.

stackhpc/18.1.0.79

Upgrade Notes

  • The playbooks under etc/kayobe/ansible have been subdivided into different categories to make them easier to navigate. This change may result in merge conflicts where playbooks have been edited downstream, and broken hooks where symlinks have been used.

    To mitigate the impact of these changes, two scripts have been added to help fix broken links. tools/get-new-playbook-path.sh returns the new category of a given playbook, and tools/magic-symlink-fix.sh attempts to fix any broken symlinks in the kayobe configuration.

stackhpc/18.1.0.78

New Features

  • Adds burn-in element to Ironic Python Agent to support Ironic burn-in capability.

stackhpc/18.1.0.77

Bug Fixes

  • Bump kayobe-automation to include fix that runs kayobe commands from the directory specified in KAYOBE_CONFIG_PATH.

stackhpc/18.1.0.73

New Features

  • Added support for building overcloud host images from Pulp package mirrors. This feature can be toggled with the flag stackhpc_build_overcloud_image_from_pulp_package_mirrors. By default, this feature is enabled in the ci-builder environment and disabled otherwise. StackHPC overcloud host images will now be built with packages from Ark.

stackhpc/18.1.0.71

New Features

  • Add definitions for NVIDIA A30 and NVIDIA H100L to the stackhpc_gpu_data for easy deployment of such GPUs.

stackhpc/18.1.0.64

New Features

  • Updated OVN to 24.03.6-22 for Rocky Linux 9 in 2025.1. For details, see the following changelogs:

stackhpc/18.1.0.61

New Features

  • Updated the Kayobe and Kolla-Ansible version checks to support pinning to branches or a specific commit.

stackhpc/18.1.0.60

Bug Fixes

  • Bumps the radosgw_usage_exporter image to v0.1.4 to fix an issue where the exporter would crash if it was unable to gather user info from the RGW.

stackhpc/18.1.0.56

Bug Fixes

stackhpc/18.1.0.55

Bug Fixes

stackhpc/18.1.0.53

New Features

stackhpc/18.1.0.52

Bug Fixes

  • This update addresses a packaging issue introduced with Ubuntu Noble. The default Apt repository sources now use .sources files in deb822 format, stored under /etc/apt/sources.list.d/ubuntu.sources. The change ensures that all build sources, regardless of format are correctly removed during cleanup.

stackhpc/18.1.0.47

Bug Fixes

  • Fixed broken panels in the project metrics Grafana dashboard by updating the Prometheus Libvirt exporter and editing the dashboard queries.

    The exporter was updated from v1.6.0 to v2.2.0. A full breakdown of changes can be found here

stackhpc/18.1.0.45

Bug Fixes

  • The RabbitMQTooMuchReady prometheus alert has been removed.

    Since migrating to durable queues, messages are sent to all receivers, but only one will respond. This results in high numbers of messages staying in the ready state. The messages will be deleted automatically once they hit resource limits. This appears to have no noticeable impact on CPU, memory, or disk usage, so the alert is not needed.

stackhpc/18.1.0.37

New Features

  • Use upstream source of Magnum for Kolla container image build. The StackHPC downstream Magnum repository was for keeping Heat driver support. As Heat support was deprecated, there’s no need to build Magnum from downstream source.

stackhpc/18.1.0.32

New Features

  • Updated the version of magnum-capi-helm driver to 1.3.0. For the list of features and fixes see its release notes.

stackhpc/18.1.0.27

New Features

  • Refreshed all container image tags for both Rocky and Ubuntu. This applies recent kolla build changes, so that aarch64 images are available for Rocky 9, and RabbitMQ now uses version 4.1.

stackhpc/18.1.0.25

New Features

  • Support OpenStack upgrade from 2024.1 to 2025.1.

  • Added RabbitMQ 4.1 support. StackHPC Kayobe Config now deploys RabbitMQ 4.1 by default.

stackhpc/18.1.0.24

New Features

  • The container-image CI workflow now builds Rocky Linux 9 images for the aarch64 architecture (in addition to amd64) and publishes a multi-architecture manifest for each tag. This enables the same image tag to be pulled seamlessly on both x86-64 and ARM64 hosts.

stackhpc/18.1.0.22

New Features

  • Bumps Ceph to Squid v19.2.3.

stackhpc/18.1.0.16

New Features

  • Merged OpenBao playbooks and Hashicorp Vault playbooks. They starts with the prefix “secret-store”. By default, the playbooks will deploy OpenBao. To use Hashicorp Vault, set stackhpc_ca_secret_store: vault.

stackhpc/18.1.0.12

New Features

  • Added support for DOCA OFED on Rocky Linux 9.6 at version 2.9.3. The package versions for Rocky 9.4 and 9.5 remain unchanged, using 2.9.1.

  • Increase the number of retries when waiting for Pulp to become ready. This is to avoid issues with Pulp taking longer than expected to start up.

  • The radosgw-usage-exporter is now put behind HAProxy. To facilitate this, stackhpc_radosgw_usage_exporter_port had been renamed to stackhpc_radosgw_usage_exporter_backend_port (it remains 9242) and stackhpc_radosgw_usage_exporter_frontend_port (defaults to 9240) has been introduced.

  • Add support of the operator supplying the rated DWPD value for NVMe drives. There is a playbook get-nvme-drives.yml that will populate a new section in the stackhpc-monitoring.yml file with drive model names for NVMes in the cloud. The operator can then fill in the rated DWPD values for each drive.

  • Add support for syncing neutron-bgp-dragent image into Pulp.

Bug Fixes

  • Fixes an issue where the external Grafana endpoint would be added to Prometheus Blackbox Exporter config, even when enable_grafana_external was disabled.

  • Fix Kayobe version checks that were failing on multiuser Ansible control hosts.

  • Ensure that the correct tag is used for OpenBao repository in Pulp.

  • Fixes an issue where object storage metrics were missing from Prometheus by putting the radosgw-usage-exporter behind HAProxy.

  • Minor issue with RabbitMQ dashboard panel showing no data with default scrape settings.

stackhpc/18.1.0.11

Bug Fixes

  • Removed egrep usage from scripts. Instances have been replaced with grep -E. This resolves the warning egrep: warning: egrep is obsolescent; using grep -E.

stackhpc/18.1.0.7

New Features

  • Added repo definitions for ProxySQL for AlmaLinux 9 in pulp. ProxySQL container images for Rocky Linux 9 are now built using version controlled packages from Ark.

stackhpc/18.1.0.5

New Features

  • The Seed Pulp container version has been bumped from 3.43.1 to 3.81.0.

stackhpc/18.1.0.2

New Features

  • Ansible dependencies have been bumped to the latest available versions. This includes:

    • stackhpc.cephadm - 1.19.1 -> 1.19.3

    • ansible-lockdown.rhel9_cis - 1.3.1 -> v1.3.4

    • geerlingguy.pip - 2.2.0 -> 3.1.0

    • monolithprojects.github_actions_runner - 1.18.5 -> 1.25.1

    • geerlingguy.docker - unpinned -> stackhpc/7.0.1.1

    • ansible-modules-hashivault - 5.2.1 -> 5.3.0

    ansible-lockdown.ubuntu22_cis has been replaced with ansible-lockdown.ubuntu24_cis, which is pinned to 1.0.1.

stackhpc/18.0.0.4

New Features

  • Added support for Rocky Linux 9.6, including host packages and a full container image refresh.

Upgrade Notes

  • 9.6 is now the default release for Rocky Linux.

stackhpc/18.0.0.1

New Features

  • Updates OpenSearch to 2.11.1.

  • Bumped pulp repo versions for Q2 2024 Bumped Kolla image tags for Q2 2024 Bumped prometheus server from 2.38.0 to 2.51.1 Bumped prometheus alertmanager from 0.24.0 to 0.26.0 Bumped prometheus blackbox exporter from 0.23.0 to 0.25.0 Bumped prometheus cadvisor exporter from 0.48.0 to 0.49.1 Bumped prometheus haproxy exporter from 0.13.0 to 0.15.0 Bumped prometheus memcached exporter from 0.10.0 to 0.14.3 Bumped prometheus msteams from 1.5.1 to 1.5.2 Bumped prometheus mtail from 3.0.0-rc50 to 3.0.0-rc53 Bumped prometheus mysqld exporter from 0.15.0 to 0.15.1 Bumped prometheus node exporter from 1.4.0 to 1.7.0 Bumped prometheus openstack exporter from 1.6.0 to 1.7.0 Bumped prometheus ovn exporter from 1.0.6 to 1.0.7 Bumped opensearch from 2.11.1-1 to 2.13.0-1 (Rocky Linux 9) Bumped opensearch from 2.12.0 to 2.13.0 (Ubuntu Jammy) Bumped grafana from 10.1.5-1 to 10.4.2-1 (Rocky Linux 9) Bumped grafana from 10.4.0 to 10.4.2 (Ubuntu Jammy)

  • Added two alerts (warning and critical) that are triggered when the ratio of (free_swap_space / total_swap_space) is below thresholds. Each threshold can be modified by altering value of alertmanager_node_free_swap_warning_threshold_ratio and alertmanager_node_free_swap_critical_threshold_ratio.

    Currently this solution has limitation of having one-size fits all policy. This can cause unwanted alerts for the hosts which utilise swap heavily Therefore it is recommended to tune the thresholds or apply silence rules for the needs.

  • Add blazar project Kolla container images. Blazar is a resource reservation service for OpenStack. Blazar enables users to reserve a specific type/amount of resources for a specific time period and it leases these resources to users based on their reservations.

  • Adds caso container images. cASO is an is an accounting reporter that supports Cloud Accounting Usage Records. For more information, see the upstream docs. Note that this container does not exist in upstream Kolla and is maintained downstream by StackHPC.

  • Adds code to the globals.yml file to add endpoints for the ceph_mgr_exporter. If ceph is configured correctly, managers will be under the mgrs inventory group. If this group is empty, then the variable will just be empty (the KA default). This also requires setting kolla_enable_prometheus_ceph_mgr_exporter to true.

  • A confirmation prompt has been added to reboot.yml to help avoid rebooting the wrong hosts by mistake. This check can be skipped by setting confirm_reboot: true.

  • Adds NVMe and S.M.A.R.T utilities to the overcloud host image built by DIB.

  • Add optional support for relabelling network devices in Prometheus. Use network names as defined in kayobe, instead of network device names. Reuse of device names within an environment is not supported.

  • Adds support for deploying GitHub runners and creating GitHub workflows for use within Kayobe Automation. Two playbooks and their requirements have been added to ansible/ in addition to the relevant groups defined with some useful default variables where appropriate. Finally, documentation has been added to cover how to deploy these runners and workflows.

  • The playbook hotfix-containers.yml has been added. This allows arbitrary files to be copied into, and/or arbitrary commands to be executed within, overcloud containers.

  • Support for Ubuntu 22.04 Jammy Jellyfish repositories have been added to the Yoga Release.

  • Adds support for Let’s Encrypt external account binding (EAB).

  • Set monitoring services be enabled by default in the ci-multinode environment.

  • Add additional Tempest tests plugins covering; Barbican, Cinder, Cloudkitty, Designate, Glance, Ironic, Keystone, Magnum, Manila, Neutron and Octavia. Add load lists for the new Tempest plugins. Use docker-rally v2.0.1.

  • Add support for deploying OpenBao across the seed and overcloud hosts for the purpose of internal and backend TLS generation.

  • Add support for highly available Raft when using OpenBao on overcloud hosts.

  • OpenSearch container images have been added.

  • Add playbook to install pre-commit hooks and register them with git. The hooks currently configured to be installed will check yaml syntax, fix new line at end of file and remove excess whitespace. This is currently opt-in which can be achieved by running install-pre-commit-hooks playbook.

  • Adds RADOS Gateway usage exporter support.

    To deploy the exporter, set the variable stackhpc_enable_radosgw_usage_exporter to true. Then run playbook deploy-radosgw-usage-exporter.yml. A certificate path needs to be set to stackhpc_radosgw_usage_exporter_cacert if internal TLS is enabled.

  • Added the rekey-hosts.yml playbook to automatically rotate the SSH keys on all hosts.

  • Add the package repository configuration required for Rocky Linux 9 support.

    Add CI for Rocky 9 hosts.

  • Added support for Rocky Linux 9.2 repositories and made 9.2 the default version.

  • Added support for Rocky Linux 9.3 repositories and Kolla containers. Made 9.3 the default version for Rocky Linux.

  • Updated Rocky Linux 9.2 pulp repo versions. Added Rocky Linux 9.3 pulp repo versions. Rebuilt Kolla containers with Rocky Linux 9.3.

  • Added support for Rocky Linux 9.4 repositories and Kolla containers. Made 9.4 the default version for Rocky Linux.

  • Updated Rocky Linux 9.3 pulp repo versions. Added Rocky Linux pulp repo versions. Rebuilt Kolla containers with Rocky 9.4.

  • Add support for a basic user for Pulp operations instead of using the admin user for usage. Can be enabled by setting pulp_stack_password.

  • Added support for Ubuntu 24.04 Noble Numbat as a host operating system. Repositories and configuration for Ubuntu Noble have been added.

  • Adds support for using a VMs as compute and controller nodes in the ci-multinode environment by dynamically setting the MTU of the networks in networks.yml and removing the static definition of the network interfaces for the compute and controller groups.

  • Add Wazuh deployment playbook.

  • Adds an alert to check that there is exactly one active router on ML2/OVS based deployments.

  • Adds utility playbooks to build and rotate amphora images. For more details check out the Octavia section of the Operator Guide included in the documentation.

  • Adds support for Ubuntu Jammy and Rocky 9 to the CIS benchmark hardening playbook: cis.yml. This playbook will need to be manually applied.

  • Adds a hook to automatically run the CIS benchmark hardening playbooks as part of host configure. This is guarded by the stackhpc_enable_cis_benchmark_hardening_hook configuration option and is disabled by default.

  • Adds kolla config merging options to the Kolla custom config generation section of etc/kayobe/kolla.yml.

  • Adds alerts for software raid failures.

  • Brings in new neutron container images to add batching support to Networking Generic Switch. This is opt in via the ngs_batch_requests configuration option and only affects Ironic deployments that use Networking Generic Switch. See the following PR for more details.

  • Adds the networking-mlnx mechanism driver to the Neutron Server container and ebrctl utility to the Nova Compute container. This allows you to use the kolla_enable_neutron_mlnx feature flag.

  • Updates neutron containers to contain a version of networking-generic-switch with support for trunk ports when using DellOS 10 or Cisco switches. See this PR for more details.

  • Updates neutron containers to contain a version of networking-generic-switch with support for DellOS 10. See this PR for more details.

  • Improvements to the ci-aio automated deployment script to allow the script to successfully run on LVM-based images.

  • Added a script to the AIO environment that can be used to quickly deploy an AIO for testing.

  • Added a custom policy to Ironic that allows users with the admin role to list all baremetal nodes. This is required at sites where baremetal provisioning targets a specific node, as we need to look up the node’s uuid to pass as the hypervisor hostname.

  • Adds time information to tasks using the ansible.posix.profile_tasks callback.

  • Adds some basic tuning of Ansible, including use of 20 forks, enabling SSH pipelining, YAML-formatted output, and disabling fact variable injection.

  • A default firewall configuration is now included on an opt-in basis. The rules are defined under etc/kayobe/inventory/group_vars/all/firewall. More information can be found here

  • Added Blackbox monitoring for backend endpoints by default. Note that this configuration will only work if the Blackbox exporters have access to the backend endpoints.

  • Prometheus Alertmanager has been updated to 0.28.1. This release includes support for Microsoft Teams notifications.

  • Updates the StackHPC Cephadm Ansible collection from 1.18.0 to 1.19.1.

  • Bump Ceph to v19.2.1.

  • Bumped Horizon kolla image Bumped Grafana from 10.1.5-1 to 10.4.2-1 (CentOS & Rocky Linux) Bumped Grafana from 10.4.1 to 10.4.2 (Ubuntu) Bumped Prometheus-msteams from 1.5.0 to 1.5.2

  • Updates Magnum CAPI Helm driver version to v0.11.0

  • Updates Magnum CAPI Helm driver version to v0.12.0

  • Updates Magnum CAPI Helm driver version to v0.13.0

  • Updated OpenvSwitch to 3.3.4-115 and OVN to 24.03.5-88 for Rocky Linux 9 in Caracal. For details, see the following changelogs:

  • Kolla Toolbox, Manila, Neutron, Nova, and Octavia containers received updates on both Rocky Linux 9 and Ubuntu. Only the Rocky Linux 9 images include the new OVS versions.

  • Upgrades the redfish exporter container image to the v2.x series.

  • Adds support for lenovo hardware to the redfish exporter dashboard.

  • Adds the stackhpcredfish_exporter_scrape_interval, stackhpc_os_capacity_scrape_interval, and stackhpc_prometheus_openstack_exporter_interval configuration variables.

  • magnum container now has capi driver

  • Bumped the base image for Ubuntu 22.04 containers.

  • Additionally bumped Nova, Neutron and Octavia images for both Rocky and Ubuntu.

  • Add kolla_ceph_conf_append configuration option to specify a string to be appended to all ceph.conf files gathered from a ceph cluster using kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/cephadm-gather-keys.yml.

  • Adds support for using Ceph HAProxy and Keepalived images stored in Pulp. This is enabled automatically if stackhpc_sync_ceph_images is set to true.

  • Adds two new custom playbooks for placing Ceph hosts into and removing them from maintenance:

    • ceph-enter-maintenance.yml

    • ceph-exit-maintenance.yml

  • Added support for the Squid release of Ceph.

  • The Cephadm pre and post commands now support default commands with the variables cephadm_commands_pre_default and cephadm_commands_post_default. As such, any extra commands should be added to the variables cephadm_commands_pre_extra and cephadm_commands_post_extra.

  • Adds a new diagnostics.yml playbook that collects diagnostic information from hosts. The diagnostics are aggregated to a directory ($PWD/diagnostics/ by default) on localhost. The diagnostics include:

    • Docker container logs

    • Kolla configuration files

    • Log files

    The collected diagnostic information contains sensitive information such as passwords in configuration files.

  • The Heat OpenStack service is now disabled by default.

  • The Ironic Prometheus Exporter is now disabled by default as redfish exporter is preferred. If you need to use the Ironic Prometheus Exporter, you can enable it by setting the kolla_enable_ironic_prometheus_exporter option to true.

  • Using DOCA LTS 2.9.1.

  • DOCA workflows updated to build kernel modules only, relying on Release Train synchronisation of DOCA upstream for userspace packages.

  • Improved documentation now including DOCA install playbook.

  • Local Pulp syncs for DOCA/DOCA kernel module repository.

  • The Docker CE package for Ubuntu has been bumped from 5:24.0.6-1 to 5:25.0.0-1 This is a side effect of separating out the repos for Docker CE for Ubuntu Jammy/Focal.

  • Adds a panel in the Hardware Overview dashboard to show DWPD (Drive writes per day) for NVMEs. This is calculated by dividing the total bytes written in the past 24 hours by the drive capacity. This is currently only supported on NVMEs.

  • Adds alerts that will fire after 1 DWPD is sustained for 7 days, and a critical alert if 1 DWPD is sustained for 30 days.

  • Provide ELRepo 9, which in turn provides packages to support be2net and mpt3sas hardware. Configuration of ELRepo 9 is disabled by default and may be enabled by setting dnf_install_elrepo_9: true.

  • Enable building of the Neutron BGP Dragent container image.

  • Mariabackup is now enabled by default.

  • The flag om_enable_rabbitmq_high_availability is now set to true. Adds tags for new RabbitMQ containers to update to RabbitMQ version 3.9.22.

  • Adds an etcd Kolla container image. This can be used for OpenStack service coordination as a tooz backend, or for batched processing of switch configuration in Networking Generic Switch (this requires a downstream NGS patch).

  • Adds drive temperatures to the table on the hardware overview dashboard and a timeseries to show the temperature over time.

  • Adds picker to hardware overview dashboard to select a specific host to show drive information for.

  • Adds a new Prometheus alert FluentdBufferTooLarge which is raised when the total size of queue buffers grows above 128 MiB.

  • Added a templated set of default Prometheus Blackbox exporter endpoints.

  • Openstack dashboard is now used as default Grafana dashboard.

  • Adds support for synchronising HashiCorp Consul and Vault images to a local Pulp registry.

  • The StackHPC overcloud host images have been rebuilt using new packages for the Caracal release.

  • Adds a new build workflow to Release Train for building IPA images, and deployment for Release Train IPA images to Bifrost and Ironic inspector.

  • Release Train IPA is enabled by default.

  • Configures the Ironic Python Agent with useful settings for inspection, such as the extra-hardware and mellanox elements.

  • ironic-prometheus-exporter container image has been added and enabled by default when kolla_enable_ironic and kolla_enable_prometheus are true (following upstream defaults).

  • StackHPC Kayobe Configuration container images for CI/CD with Kayobe Automation are now published to GitHub Container Registry (GHCR) at ghcr.io/stackhpc/stackhpc-kayobe-config. The image is tagged with the name of the release branch, e.g. stackhpc/yoga.

  • Workflow to update Kolla dependencies (Kayobe, Kolla and Kolla-Ansible) to the latest tag available in the StackHPC branch via CI.

  • Added playbooks to check the installed Kayobe/Kolla-Ansible versions against the expected versions in Kayobe configuration. These checks will run on Kayobe bootstrap, host and service operations.

  • Adds a new variable stackhpc_pulp_sync_for_local_container_build which, when set to true, configures the local Pulp server to sync all package repositories required for building kolla containers on a local kolla build host.

  • Rocky images have been rebuilt and are now based on Rocky 9.3.

  • Allow Kolla dependency updates on non-default branches.

  • Enable TLS for the Seed Pulp service. Set pulp_enable_tls: true and provide paths to a TLS certificate and key using pulp_cert_path and pulp_key_path respectively.

  • The Openstack Dashboard in Grafana now includes logs from Openstack services.

  • Adds a standard LVM configuration that is compatible with the new overcloud host image.

  • adds helm client into magnum container

  • Adds support for Manila in the ci-multinode environment using the CephFS native backend. This is disabled by default, but can be enabled by setting the following variables in the kayobe configuration: kolla_enable_manila: true kolla_enable_manila_backend_cephfs_native: true

  • Updated the documentation for the ci-multinode to include instructions on how to set up and test Magnum.

  • Added support for Wazuh in the ci-multinode environment.

  • Adds a new Prometheus alert HostNetworkBondDegraded which will be raised when at least one bond member is down.

  • Adds a new Prometheus alert HostNetworkBondSingleLink which will be raised when a bond is configured with only one member. This can happen when NetworkManager detects that a bond member is down at boot time. This alert can be disabled by setting alertmanager_warn_network_bond_single_link to false.

  • Neutron containers are now built from our StackHPC fork.

  • Updates Prometheus Node exporter to version 1.5.0.

  • Adds NTP alerts to prometheus alertmanager.

  • Nvmemon now reports physical size of the disk.

  • Adds alerts for Octavia load balancers and amphorae. Alerts are triggered when load balancers enter the ERROR or DEGRADED states, or when amphorae enter the ERROR state.

  • Adds a new Grafana dashboard for Octavia. This dashboard is used to monitor the load balancers as well as the amphorae.

  • Implement an OFED workflow that builds kernel modules to support OFED drivers in release train kernels and upload OFED kernel/userspace drivers to Ark.

  • This patch adds OpenStack Capacity metrics and exporters to StackHPC Kayobe Config. This includes a deployment playbook, Prometheus scrape jobs and HAProxy configurations to support this change.

  • Adds support for providing a CA certificate for OpenStack Capacity exporter.

  • Automatic deployment for OpenStack Capacity via a Kayobe service deploy hook using kolla admin credentials.

  • Per OSD usage metrics are now available in the OSDs dashboard. The dashboard now includes a new section that displays a histogram of of the utilization of each OSD in the cluster. This can be useful for identifying OSDs that are outliers in terms of utilization, and may need to be rebalanced. Additionally, there is a histogram displaying the usage of the bluestoreDB for each OSD.

  • Adds a standard overcloud Diskimage Builder (DIB) host image configuration.

  • Adds ethtool and pciutils to the overcloud host disk image.

  • OVN version in Rocky Linux 9 container images has been updated to 24.03 (latest LTS).

  • Add support for deploying OVN SB Relays, this can be enabled by setting kolla_enable_ovn_sb_db_relay in kolla.yml to true. To use this feature stackhpc/16.4.0.5 version of stackhpc/kayobe and stackhpc/18.4.0.6 version of stackhpc/kolla-ansible is required.

  • Open vSwitch has been pinned to the latest LTS release (3.3) in Ubuntu Noble container images.

  • Added templates and a playbook to simplify configuration of PCI passthrough GPUs. GPU types can be mapped to inventory groups with the gpu_group_map variable, which will configure the host and Nova automatically. A list of supported GPUs can be found in etc/kayobe/stackhpc-compute.yml under stackhpc_gpu_data.

  • Removed custom Kolla configuration files that have since been updated upstream.

  • Removed Kolla build configuration pins to use versions from upstream. This change affects prometheus-v2-server (2.54.1 -> 2.55.1), and prometheus-memcached-exporter (0.14.4 -> 0.15.0). prometheus-blackbox-exporter has also been unpinned, but the version remains the same.

  • All Ansible dependencies are now pinned to specific versions. The stackhpc.vxlan role is pinned to 1.1.0, and ansible-role-docker is pinned to stackhpc/7.0.1.1.

  • Bumps the Prometheus container images to bring in Prometheus v3.

  • Prebuilt overcloud host images can now be pulled from Ark using the stackhpc_download_overcloud_host_images variable. The image is selected based on os_distribution and os_release.

  • Adds a custom playbook (pulp-auth-proxy.yml) for deploying an authenticating proxy for Pulp. This can be used when building container images to avoid leaking credentials for package repositories into the built images or their metadata.

  • Allows to synchronise a custom list of containers to Pulp using the stackhpc_pulp_repository_container_repos_extra and stackhpc_pulp_distribution_container_extra variables.

  • Added a new playbook pulp_sync_publish_promote that can be used to sync, publish and promote all repositories in a single step, as well as sync and publish container repos. If you do not want to promote repos then run with -e repo_promote_production=false.

  • Raises an alert when the count of RabbitMQ ready messages increases above a threshold.

  • Added a script to automate RabbitMQ quorum queue migrations.

  • Bumps the RabbitMQ container image tag to upgrade RabbitMQ to v4.0

  • Adapt threshold of RabbitMQ connection alert based on the size of the deployment to avoid spurious alerts.

  • Added a new script, rabbitmq-queue-migration.sh, which will migrate to the new RabbitMQ durable queues. This is intended for use prior to an upgrade to Epoxy.

  • Re-enable Pulp Ubuntu repositories.

  • Adds support for deploying a Prometheus Redfish exporter container on the seed. This can be used to query the overcloud BMCs via their redfish interfaces to produce various metrics relating to the hardware, and system health.

  • Package repositories and container images for CentOS Stream based deployments have been updated. Key packages to note are:

    • Kernel

      • version: 4.18.0

      • release: 448.el8

    • Libvirt

      • version: 8.0.0

      • release: 6.module_el8.7.0+1140+ff0772f9

    • OVS

      • version: 2.17.0

      • release: 71.el8s

    • OVN

      • version: 22.09.0

      • release: 11.el8s

  • Container images for Ubuntu based deployments have been updated. Key packages to note are:

    • Libvirt

      • version: 8.0.0

      • release: 1ubuntu7.4~cloud0

    • OVS

      • version: 2.17.3

      • release: 0ubuntu0.22.04.1~cloud0

    • OVN (unchanged since last container build)

      • version: 22.03.0

      • release: 0ubuntu1~cloud0

  • Disable the CIS hardening rule *_rule_5_4_3_2 to prevent TMOUT from being applied which can disrupt a development environment as it closes TMUX panes and servers and may close active ssh session.

  • Rocky Linux 9 image has been rebuilt with missing base packages (e.g. microcode_ctl) by installing ‘Minimal Install’ DNF group. Also cloud-init from CentOS 9 Stream has been installed with NetworkManager support.

  • Added support for Rocky Linux 9.5, including host packages and a full container image refresh.

  • Made 9.5 the default release for Rocky Linux.

  • Sync Rocky Linux 8.7 RPM repositories to local Pulp servers.

  • Enables SMART monitoring. Manual action is required, please see the monitoring documentation for the procedure.

  • The smartmon-tools playbook now ensures that the cron service is running as in some cases it may not be running by default.

  • Split cephadm_commands into cephadm_commands_pre and cephadm_commands_post commands. This allows the user to run commands that must be run before the rest of the post-deployment configuration, as well as commands that rely on resources created by the post-deployment config.

  • Adds a new stackhpc-openstack-tests.yml playbook that executes tests in the StackHPC OpenStack Tests repository. Both the playbook and tests are currently experimental, and are currently targeting only an all-in-one CI use case.

  • Added a new group variable - stackhpc_repos_enabled - for unified control over usage of StackHPC Release Train package repositories. This makes it easier to set which hosts do or do not pull packages from release train.

  • Added the stop-openstack-services.yml playbook, which can be used to stop OpenStack services across the overcloud.

  • Supports adding CA certificates to the Tempest container trust store.

  • The default Tempest concurrency has been increased from 2 to 16. This is often easily achievable in production systems.

  • OVN has been pinned to the latest LTS release (24.03) in Ubuntu Noble container images.

  • Refreshed all Ubuntu host package versions and contianer images for December 2024.

  • Use the StackHPC fork for building Blazar images with customizations to support flavor-based reservation.

  • Updates Grafana to 9.4.7 version.

  • The HAProxy dashboard and alerts have been updated to use the new metrics source in Kolla Ansible. For further details see here.

  • Upgrades kayobe-automation submodule to 7676aa8.

    Upgrades kayobe-workflows collection to v1.1.0.

    Kayobe-automation config-diff now runs in parallel and generates both the old and new configuration at the same time. This should improve config-diff wait times.

    Add support for the pulp-sync-content run book.

  • Updates Magnum CAPI Helm driver version to OpenDev v1.0.0

  • Updated OpenvSwitch to 3.3.0-88 and OVN to 24.03.5-14 for Rocky Linux 9 in Caracal. For details, see the following changelogs:

  • Kolla Toolbox, Manila, Neutron, Nova, and Octavia containers received updates on both Rocky Linux 9 and Ubuntu. Only the Rocky Linux 9 images include the new OVS versions.

  • Updates Prometheus to version 2.55.1.

  • Upgrades Pulp from 3.21 to 3.22.

  • Disables Pulp analytics.

  • Sets Pulp worker based on available CPU cores. This may improve performance when pulling container images to many hosts simultaneously.

  • Upgrades Pulp from 3.22 to 3.23.

  • Upgrades Pulp from 3.23 to 3.24.

  • Restrict the content that is synced to the client by using include tags. This feature ensures that the tags as defined within kolla-image-tags.yml are synced.

  • Allow for easy customisation of the number of expected RabbitMQ nodes when evaluating the alert RabbitMQNodeDown. It is set by the alertmanager_number_of_rabbitmq_nodes which defaults to the number of controllers. This is benefical for deployments that do not use a standard three node setup.

  • Adds support for package repository snapshots via Pulp. A local Pulp server is deployed on the seed, which syncs package repositories and container images from the StackHPC Ark Pulp server. Control plane servers pull packages and container images from the local Pulp server.

  • The EPEL package repository is disabled by default. It may be enabled by setting dnf_enable_epel to true.

  • Uses StackHPC source code repositories for kolla, kolla-ansible, and bifrost.

  • Supports Kolla CentOS Stream 8 source container images.

  • Adds custom playbooks for compute host maintenance:

    • nova-compute-drain.yml

    • nova-compute-disable.yml

    • nova-compute-enable.yml

    • reboot.yml

  • Upgrades the version of wazuh-ansible to v4.10.0. This brings in the SCA CIS checks for Rocky Linux 9 by default.

  • Wazuh can now de deployed with additional custom SCA policies. Just add the policy file(s) to the directory {{ kayobe_env_config_path }}/wazuh/custom_sca_policies.

  • Adds a custom playbook to reset the RabbitMQ cluster and restart OpenStack services that use it, rabbitmq-reset.yml.

  • Adds a custom playbook to configure swap, swap.yml.

  • Adds the Kayobe Automation Git repository as a submodule, and provides some basic configuration for it in an .automation.conf directory.

  • Adds support for deploying a Squid caching proxy as a custom container on the seed.

  • Enables Elasticsearch, Grafana, Kibana, Prometheus by default. Provides standard dashboards for Grafana and alerting rules for Prometheus.

  • Bumped Horizon kolla image Bumped Grafana from 10.1.5-1 to 10.4.2-1 (Rocky Linux) Bumped Grafana from 10.4.1 to 10.4.2 (Ubuntu) Bumped Prometheus-msteams from 1.5.1 to 1.5.2

Known Issues

  • Generate backend TLS files for network hosts. This fixes backend TLS configuration for deployments where some API services are running on network hosts.

  • Backend Blackbox monitoring will not work if the exporter does not have access to the backend OpenStack endpoints. This usually happens when separate monitoring nodes are deployed. In this case, move the Blackbox exporter to the Haproxy group, remove the endpoints from etc/kayobe/kolla/inventory/group_vars/prometheus-blackbox-exporter, or silence the alerts permanently.

  • Magnum has been pinned to the Antelope release due to an issue upstream in Caracal. See here for more details.

Upgrade Notes

  • Ensure that your deployment has only one nova-compute-ironic service running per conductor group. See the operations / nova-compute-ironic doc for further details.

  • Kolla config merging is enabled by default in the Antelope release of Kayobe. This was quite an extensive change and whilst backwards compatbility was one of the goals, there may be some situations where refactoring of your Kolla config will be necessary. Extra care should be taken if you are using the multiple environments feature. It is recommended that you carefully check the diff in the resultant Kolla configuration by following these steps to check for missing config or duplicated config options. The kolla_openstack_custom_config_environment_merging_enabled option can be set to False to revert back to the old behaviour.

  • Bifrost Ironic debug logging is now disabled by default. Change ironic_debug to true to revert.

  • The prometheus_blackbox_exporter_endpoints_custom variable has been renamed to stackhpc_prometheus_blackbox_exporter_endpoints_custom to avoid conflits with an upstream Kolla-Ansible var of the same name.

  • Rebuilt all kolla and package repo tags to bring in kernel fixes and apply CentOS image build customisations that were previously being ignored.

  • CentOS Stream 8 snapshots have been bumped and new container images are available. Make sure to sync these into your local pulp. The yum repositories must be reconfigured to exclude a buggy version of iptables. To do this use: kayobe overcloud service reconfigure -kt none -t dnf.

  • CentOS Extras has been replaced with CentOS Extras Common. You may need to use the --allowerasing option with DNF if you have packages installed from the old repo. This is a one time only thing and on the next package update you can drop this argument.

  • Updates default Ceph images to v17.2.7 for Quincy.

  • Bumped focal package versions due to unmet depenencies

  • Updates Consul to 1.16.4 and Vault to 1.14.8.

  • Updates Consul to 1.16.3 and Vault to 1.14.6.

  • Updates Magnum CAPI Helm driver version to v0.11.0

  • Bumps octavia container versions

  • Bumped rocky 9 package versions due to missing snapshot

  • Increases default os_capacity_scrape_interval to 5m. If you already customise this please move to the new stackhpc_os_capacity_scrape_interval variable.

  • container tags for magnum capi changes

  • Updates the stackhpc.cephadm collection to version 1.18.0.

  • Updates Ceph Pacific container image to v16.2.11.

  • Automatically install Quincy if the node is running Ubuntu 22.04, else install Pacific.

  • Bumps the default Ceph Reef container image to v18.2.7.

  • Updates the default version of Ceph to Reef. The following container tags are used for Ceph:

    • ceph: v18.2.4

    • haproxy: 2.6

    • keepalived: 2.2.4

  • Squid has become the default release of Ceph. The new default image tag is v19.2.0.

  • Increase stackhpc.cephadm collection to version 1.12.2.

  • To match the new CIS benchmark defaults on Ubuntu, you should remove the ipv6.disable=1 kernel command line option. If you wish to carry on with the current settings, change ubtu22cis_ipv6_required to false.

  • The Heat service is now disabled by default. This behaviour can be overriden by setting kolla_enable_heat: true in etc/kayobe/kolla.yml. It is recommended that you migrate to the CAPI Helm driver for Magnum wherever possible.

  • Enables Docker live restore by default. This may be disabled by setting docker_daemon_live_restore to false in docker.yml.

  • Support for Rocky Linux 9.1 to 9.4 has been dropped. Hosts must be updated to RL9.5 or above before upgrading to this OpenStack release.

  • The flag om_enable_rabbitmq_high_availability is now set to true. As this enables durable queues, RabbitMQ will need to be reset, and the services which use it restarted. Tags are added to update the RabbitMQ containers to version 3.9.22.

  • Enabled ML2/OVN by default. Checks preventing accidental migration from ML2/OVS were added in Kolla Ansible. If you are using a Neutron plugin other than ML2/OVN, set kolla_enable_ovn to false.

    OVN distributed FIP is disabled, to enable it set neutron_ovn_distributed_fip to true in etc/kayobe/kolla/globals.yml.

  • Updates the Ansible configuration to fail on any unparsed inventory source. If you are using a separate Ansible configuration for Kolla Ansible, you may wish to add this setting in etc/kayobe/kolla/ansible.cfg.

  • Change default values for DIB_GRUB_TIMEOUT_STYLE and DIB_GRUB_TIMEOUT. The default value for DIB_GRUB_TIMEOUT_STYLE will be menu and for DIB_GRUB_TIMEOUT will be 5. Adding ConfigDrive to DIB_CLOUD_INIT_DATASOURCES var list to fix cloud-init issue.

  • The overcloud host image build workflow now uploads the built image to SMS as well as ARK, allowing it to be tested both manually and through AIO CI jobs.

  • Kolla Ansible and Kayobe version checks are enabled by default which may fail on existing deployments using custom forks or branches for Kayobe and Kolla-Ansible. To disable version checks in configuration set stackhpc_enable_kayobe_check and stackhpc_enable_kolla_ansible_check to false.

  • The path used to store Wazuh certificates has changed. local_certs_path is now set to the environment directory e.g $KAYOBE_CONFIG_PATH/environments/<environment>/wazuh or $KAYOBE_CONFIG_PATH/wazuh/ if not using environments. The contents of $KAYOBE_CONFIG_PATH/ansible/wazuh/certificates should be moved to the new location and the empty directory should be removed.

  • The local_custom_certs_path variable has been removed. Custom wazuh certificates should be moved to $KAYOBE_CONFIG_PATH/environments/<environment>/wazuh/wazuh-certificates/ if using environments, or $KAYOBE_CONFIG_PATH/wazuh/wazuh-certificates if not.

  • Configure Nova to use more modern ‘q35’ libvirt machine type rather than ‘pc’ which is considered legacy.

  • To deploy the OpenStack Capacity Grafana dashboard, you must define OpenStack application credential variables: secrets_os_capacity_credential_id and secrets_os_capacity_credential_secret as laid out in the ‘Monitoring’ documentation.

    You must also enable the stackhpc_enable_os_capacity flag for OpenStack Capacity HAProxy and Prometheus configuration to be templated.

    You may also change the default authentication URL from the kolla_internal_fqdn and change the default OpenStack region from RegionOne with the variables: stackhpc_os_capacity_auth_url and stackhpc_os_capacity_openstack_region_name.

    To disable certificate verification for the OpenStack Capacity exporter, you can set stackhpc_os_capacity_openstack_verify to false.

  • OpenStack Capacity no longer uses application credentials. Please delete any previously generated application credentials.

  • Updated OVN package version from 22.06 to 22.09.

  • openvswitch version has been updated to ~2.17.5 on all distributions (CentOS/Rocky9 are two patches ahead of 2.17.5). Images include fixes for CVE-2023-1668.

    Ubuntu repository versions for focal and ubuntu cloud archive have been updated to 20230515.

  • Instance labels in prometheus now use inventory hostnames rather than IPs.

  • The reboot.yml custom Ansible playbook now defaults to reboot only one host at a time. Existing behaviour can be retained by setting ANSIBLE_SERIAL=0.

  • Kolla tag overrides have been refactored to allow kolla-ansible to resolve them individually by host. This means that mixed clouds can be deployed which allows for migration between distributions.

  • The swap.yml custom playbook has been removed in favour of Kayobe’s support for configuring swap. See the Kayobe documentation for details.

  • Enables SELinux in permissive mode in the overcloud host image. This matches the default configuration for SELinux in StackHPC Kayobe Configuration.

  • Dont pull apt packages from pulp for Ubuntu Jammy until Jammy packages are published.

  • Dont pull ceph packages from ceph official repos for Ubuntu Jammy until Jammy packages are published.

  • Updates the Ansible Lockdown roles for Ubuntu and Rocky Linux to 1.4.1 and 1.3.1 respectively. See UBUNTU22-CIS and RHEL9-CIS for release notes.

  • Updates the smartmon-tools.yml playbook to ensure that cron is installed before attempting to configure crontab.

  • Update Ubuntu Jammy Zed Kolla container tags.

  • Updated Ubuntu package repository versions.

Deprecation Notes

  • Hashicorp Vault for TLS generation is deprecated in favour of OpenBao. The openbao role is now used to deploy OpenBao on the seed and overcloud hosts. New deployments should use OpenBao for TLS generation. Existing deployments using Hashicorp Vault for TLS generation should be migrated to OpenBao once migration steps are available.

  • Support for Rocky Linux 9.1 to 9.4 has been dropped.

  • Disabled building of Kolla container images for Skyline

  • Kayobe-automation will now automatically detect vaulted files for the purpose of config-diff therefore, KAYOBE_CONFIG_SECRET_PATHS_EXTRA and KAYOBE_CONFIG_VAULTED_FILES_PATHS_EXTRA are no longer used

Critical Issues

  • Fixes CVE-2024-32498 with updated container images for Cinder, Glance and Nova services.

  • Disables password expiration and inactivity policies. This caused the kayobe and kolla service accounts to be locked out of the system. You should re-apply the CIS benchmark hardening playbook as soon as possible to avoid being locked out of your system.

  • Fixes CVE-2024-40767 with updated container images for Nova services.

Security Issues

  • Fixed CVE-2023-31047, CVE-2023-23969, CVE-2023-24580, CVE-2023-36053, CVE-2023-46695, CVE-2023-30861, CVE-2022-4899. CVE-2024-1135, GHSA-2m57-hf25-phgg, CVE-2023-0286, CVE-2023-50782, CVE-2024-26130 for openstack services.

    Fixed CVE-2022-41723, CVE-2023-39325 (except prometheus-alertmanager, prometheus-msteams-exporter, prometheus-haproxy-exporter, prometheus-openstack-exporter. No patch available.), CVE-2021-43565, CVE-2022-27191, CVE-2022-27664, CVE-2021-38561, CVE-2022-21698, CVE-2021-4238, CVE-2022-40083, CVE-2022-41721, CVE-2021-33194, CVE-2023-2253, CVE-2023-27561, CVE-2023-28840, CVE-2024-21626, CVE-2022-32149, CVE-2023-45142, GHSA-m425-mq94-257g for prometheus server and exporters except prometheus-libvirt-exporter and prometheus-haproxy-exporter. (Source repository of each are archived and no longer maintained)

    Fixed CVE-2023-39325, CVE-2023-45142, CVE-2023-47108, CVE-2023-49568, CVE-2023-49569, GHSA-9763-4f94-gfch, GHSA-m425-mq94-257g for grafana.

    It is advised to redeploy service with current version of images from StackHPC Release Train.

  • Rebuilt Neutron containers to include fixes for OSSA-2024-005. The vulnerability allows unauthorised users to change tags on networks in Neutron.

  • Bumps CentOS Stream 8 snapshots to include fixes for Zenbleed (CVE-2023-20593) and Downfall (CVE-2022-40982). It is recommended that you update your OS packages and reboot into the kernel as soon as possible.

  • Update Horizon on Ubuntu to include apache2 package 2.4.52-1ubuntu4.8 which fixes CVE-2023-31122.

  • Fixed CVE-2023-31047 for Horizon. Fixed CVE-2023-49569 for Grafana. Fixed CVE-2022-40083 and CVE-2021-4238 for Prometheus-msteams.

  • The Rocky 8 minor version has been bumped to 8.8 and new snapshots have been created to include fixes for Zenbleed (CVE-2023-20593), Downfall (CVE-2022-40982). It is recommended that you update your OS packages and reboot into the kernel as soon as possible.

  • The snapshots for Rocky 9.2 have been refreshed to include fixes for Zenbleed (CVE-2023-20593), Downfall (CVE-2022-40982). It is recommended that you update your OS packages and reboot into the kernel as soon as possible.

  • Bumps Ubuntu repository snapshots and container images to bring in latest security patches. This includes the microcode to patch Downfall (CVE-2022-40982). Zenbleed (CVE-2023-20593) was patched in the previous snapshot bump. To apply the microcode updates, it is recommended to reboot each host after upgrading all of the packages.

  • Kolla container images created using the stackhpc-container-image-build.yml workflow are now automatically scanned for vulnerablilities.

  • Fixes CVE-2024-44082 with updated container images for Ironic services. Note that Ironic Python Agent images also need to be updated to fully fix this vulnerability. If this is not possible, a new configuration option [conductor]conductor_always_validates_images is available. See the OSSA-2024-003 description for more details.

  • Addresses critical vulnerability CVE-2024-36039 by bumping the PyMySQL library to 1.1.1 in all affected Kolla images. This vulnerability allows SQL injection through untrusted JSON objects.

  • The Heat container images are rebuilt with yaql 3.0.0 to include patch for vulnerability OSSN/OSSN-0093. It is recommended that you redeploy Heat services in your system with the current version of Heat images from StackHPC Release Train.

  • Updates the Rocky Linux 9 SIG Security Common repository to address CVE-2024-6409 in OpenSSH.

  • Enables the Rocky Linux 9 SIG Security Common repository, which provides updated OpenSSH packages addressing CVE-2024-6387 (regreSSHion). Other packages available in this repository are currently ignored.

  • Adds a custom Apt repository to address CVE-2024-6387 in OpenSSH.

  • The upgraded kayobe-workflows collection increases the version of various Actions and containers used within GitHub based workflows, including increasing Docker in Docker to version 27.3.1 thus removing the vunerabilities present in 24.0-git.

  • Fixed CVE-2023-31047 for Horizon. Fixed CVE-2023-49569 for Grafana. Fixed CVE-2022-40083 and CVE-2021-4238 for Prometheus-msteams.

Bug Fixes

  • Added NetworkManager-config-server package to Rocky Linux 9 deployment image. Which prevents NetworkManager from automatically running DHCP on unconfigured ethernet devices and allows connections with static IP addresses to be brought up even on ethernet devices with no carrier.

  • The grafana image now includes the gnocchixyz-gnocchi-datasource and the grafana-opensearch-datasource plugins, which are the default upstream plugins.

  • Adds basic support and a document explaining how to migrate to a single nova-compute-ironic instance, and how to re-deploy the instance to another machine in the event of failure. See the operations / nova-compute-ironic doc for further details.

  • Fixes an issue where the IPA images in Ark could not be downloaded by Bifrost, due to missing authentication parameters.

  • Updates the nova-compute-ironic container image to fix bug 2019977, which would cause Ironic instances to fail to delete.

  • Fixed a syntax error in Prometheus SMART monitoring rules.

  • Rebuild and bump the Bifrost container for Xena to include fix for Error while running update_to_latest_versions: ‘’BIOSSetting’’ object has no attribute during Ironic database migrations on upgrade

  • Fixes Ceph bug #66389 causing Ubuntu Noble hosts to fail to populate OSDs, by upgrading to Ceph v19.2.1.

  • Fix bug 2086675: Performance regression for Glance with RBD backend.

  • Updates Magnum CAPI Helm driver version to v0.10.0

  • Fixes various issues with the redfish exporter dashboard.

  • Caps the number of Pulp API and content workers to 32 each to avoid errors on hosts with many CPUs.

  • Updated the version of magnum-capi-helm used in Magnum containers. This resolves an issue stopping non-default node groups from being deleted. See #2095539 for more details.

  • Fixes an issue with idempotency in the stackhpc.ceph.cephadm_keys plugin.

  • Updates Cinder container images to fix bug 1823445 (cinder.exception.MetadataCopyFailure).

  • IPV6 is no longer disabled by default in the Ubuntu CIS hardening. If using the old behaviour you may hit 2071443.

  • The CIS hardening scripts no longer change permissions of log files by default. It is preferred to configure these permissions at source i.e on whatever is creating the files. It also suffered from a time-of-check to time-of-use race condition. If you want the old behaviour you can change rhel9cis_rule_4_2_3 and/or ubtu22cis_rule_4_2_3 to true.

  • Fixes the bulk API of CloudKitty so that it now supports the migration from Elasticsearch to OpenSearch.

  • Disabled custom APT configuration for non-overcloud hosts (Ubuntu Only). This resolves the issue of the seed hypervisor attempting to pull packages from the repository on the seed before it has been deployed.

  • Miscellaneous issues with the package-build-ofed workflow are resolved in this patchset.

  • Separated out repos for Docker CE for Ubuntu Jammy/Focal. This fixes a Pulp sync issue where two “identical” repository versions existed with different checksums.

  • Fix Kolla container image build workflow failing to find default sources.list.ubuntu. The default sources.list for ubuntu now has each for Ubuntu Jammy and Noble. This upstream change was brought by Ubuntu 24.04 support for Caracal.

  • Bumps Bifrost container image tags to fix bug 2072550 which adds leading and trailing whitespace for inspection_callback_url in httpboot config when templating.

  • Fixed incorrect Opensearch Dashboards Prometheus Blackbox Exporter configuration.

  • Fix some broken links in the docs.

  • Fixes Grafana panel of top Ceph pools by capacity used. This panel was only showing the most used pool instead of as many pools as configured with the $topk variable.

  • Fixes an issue where the ceph-rgw endpoints were added to the Prometheus blackbox exporter when the Kolla loadbalancer was not in use for RGWs.

  • The Ceph version is now determined by os_release, rather than Ansible facts. Using Ansible facts caused playbooks to fail when facts are not gathered.

  • The OpenSearch backend for CloudKitty has been fixed, so the Horizon Rating panels work again.

  • Fixes an issue where the fix-houston.yml playbook would fail to reload systemd as the notify statement was incorrect.

  • Fix missing bracket in fluentd OpenSearch configuration causing container crashes when caso is enabled.

  • Fixes an issue with the growroot playbook where disks such as ‘sdp’ would become ‘sd’ due to the removal of the trailing ‘p’ when dealing with nvme devices.

  • Fixes the hardware overview dashboard to use the correct metric for displaying drive temps. Now uses an or to display whichever metric is compatible with the drives in the system. The two metrics are temperature_case_raw_value and temperature_celsius_raw_value.

  • Updates Horizon container images to fix bug 2055784, which would cause Horizon to be unable to retrieve resource providers statistics in deployments with VGPU resources.

  • Adds a custom fix-houston.yml playbook to address dmesg errors, specifically: “tc mirred to Houston: device bond0-ovs is down”. This error typically appears when OVS HW offloading is enabled, often in conjunction with VF-LAG and ASAP^2. Detailed usage instructions are provided within the playbook’s comments. Additional context is available at the following links: LP#1899364 Kernel Patch

  • Restores the use of extra IPA collectors (including extra-hardware) when using StackHPC IPA images.

  • Fixes an issue where setting redfish_exporter_scrape_group to a value other than overcloud would exclude those nodes from the redfish exporter scrapes.

  • Fixes Neutron so that load balancer FIPs are not broken on Neutron restart. See Neutron bug report.

  • Fixes the issue with using SAML2 federation in Keystone against NetIQ IdP.

  • Fixes an issue when live migrating instances to hosts with cgroups v2 enabled (Ubuntu Jammy and Rocky 9). See Nova bug report.

  • Fixes an issue with local image builds where kolla_tag had not been set. The error had the signature:

  • Fixes internet connectivity for VMs deployed in the ci-multinode environment.

  • Bumps Neutron container image tags to fix bug 2068644 which could prevent associating floating IPs with OVN-based load balancers.

  • Update neutron container images to apply keepalived PID clean up fix With this change, Neutron always deletes stale PID file if exists.

  • Fixes creation and failover of Octavia TLS-terminated load balancers when storing the certificate and key as a PKCS12 bundle in Barbican.

  • Bumps OpenSearch heap size to 8 GB, to be identical to Elasticsearch.

  • Changed the Prometheus job name of OS Capacity exporter to os_capacity which is what Azimuth is expecting to have for cloud metrics dashboard.

  • Fixes creation of over 1TB memory VMs on AMD with IOMMU enabled on Rocky Linux 9.

  • Fixes possible templating error with PCI passthrough configuration.

  • Fixes a race condition when launching multiple Ironic instances in parallel (as is commonly triggered when using Terraform/OpenTofu). See Nova bug report.

  • Bumps the radosgw_usage_exporter tag to fix an issue where duplicate metrics could be presented to Prometheus if S3 store usage was particularly high.

  • Fixes issue where Netmiko devices were sending no commands to the switch since plug_bond_to_network is overridden in networking_generic_switch/devices/netmiko_devices/init.py and PLUG_BOND_TO_NETWORK to set to None. See NGS bug report.

  • Fixes a permission issue with node exporter’s textfile collector metrics for SMART monitoring of disk drives.

  • Fixes the smartmon script to be case insensitive when checking for the inital SMART info. This is to ensure that the script works correctly on systems where the output of smartctl -i is not capitalised as previously expected by the script. This leads to badly formatted .prom files which lead to node_exporter failing to scrape the file.

  • Fixes an issue where Squid proxy could be unable to reach external servers due to a preference of choosing IPv6 connectivity by default.

  • Bumps the stackhpc_ubuntu_jammy_overcloud_host_image_version to fix an issue where authorized_keys for the ubuntu user was not populated on newly provisioned nodes. Ubuntu package snapshots are also bumped to match those used in the new host image.

  • Fixes an issue where HashiCorp Vault standby nodes would trigger a Prometheus alert. To apply this fix to an existing system, the HAProxy configuration for Vault (kolla/config/haproxy/services.d/vault.cfg) must be manually updated following the Vault documentation.

  • When using custom SCA policies for Wazuh, the agents are now correctly configured to allow commands to be executed from the manager.

  • Fixes the InstanceDown alerting rule wait time to be consistent with the alert message. The alert message says “for 5 minutes” but the rule was set to wait for 1 minute.

  • Fixes a file descriptor leak in networking-mlnx which prevented VMs using Infiniband virtual functions from provisioning after a period of time.

  • Fixes KeyError: ip_version in networking-mlnx when used in conjuction with OVN mechanism driver.

  • Fixes a regression when using growroot.yml and software raid where the playbook would fail to identify the correct disk.

  • Updates nova image to bring in a fix for parsing mdev uuids when using libvirt>=7.7. See bug for more details.

  • Fix an issue with the OSD summary pie chart not showing any data.

  • Fixes display of the OpenSearch cluster health in Grafana when in yellow state.

  • Grafana now refuses to load AngularJS plugins. As such the grafana-piechart-panel and gnocchixyz-gnocchi-datasource plugins have been removed from the Grafana image.

  • Fix Grafana HAProxy dashboard when non-default Prometheus instance labels are used.

  • Updates the stackhpc.hashicorp Ansible collection to 2.5.0. This brings in an idempotency fix for generating certificates.

  • Prevents raising a Ceph PgsUnclean alert because of backfilling which can frequently happen because of normal rebalancing activities, such as use of the Ceph balancer or OSD addition.

  • Fixes the Prometheus Blackbox Exporter backend endpoint for ironic-inspector, as this service does not support backend TLS.

  • Bumps Ironic containers to patch CVE-2024-44082, see OSSA-2024-003 for details.

  • Add unit to LowMemory alert description.

  • Fixes Octavia health monitors not being created on cluster spawn.

  • Fixes CoreDNS for Magnum clusters crashing on startup.

  • Allows cinder-csi nodeplugin to start on the same Magnum cluster host as cinder-csi controllerplugin.

  • Corrects ClusterRole rules for Magnum cluster-autoscaler, and sets cluster-autoscaler pods to use hostNetwork.

  • Fixes appending to ca.crt in make-cert-client.sh causing multiple identical ca certs being added into /etc/kubernetes/certs/ca.crt.

  • Disables metadata proxy over IPv6 inside Neutron DHCP agent to work around bug 1953165.

  • Updates the nova-compute container image to fix bug 2091033. This bug would cause nova-compute to freeze, which would result in frequent monitoring alerts.

  • Fix for nova resize API not parsing the new flavor on resize - bug 1805969.

  • Fix creation of VM instances with UEFI enabled and Secure Boot disabled.

  • Previously switchdev capabilities should be configured manually by a user with admin privileges using port’s binding profile. This blocked regular users from managing ports with Open vSwitch hardware offloading as providing write access to a port’s binding profile to non-admin users introduces security risks. For example, a binding profile may contain a pci_slot definition, which denotes the host PCI address of the device attached to the VM. A malicious user can use this parameter to passthrough any host device to a guest, so it is impossible to provide write access to a binding profile to regular users in many scenarios.

    This patch fixes this situation by translating VF capabilities reported by Libvirt to Neutron port binding profiles. Other VF capabilities are translated as well for possible future use. LP#2008238. LP#2020813.

  • Neutron ovn db sync operation will no longer removes OVN metadata ports in networks with Octavia OVN Load balancers health monitors. A maintenance task process has been added to update the existing OVN LB HM ports to the new behaviour defined. Specifically, the “device_owner” field will be updated from network:distributed to ovn-lb-hm:distributed. Additionally, the “device_id” will be populated during update action. LP#2038091.

  • Restores valid value for the flavor_id label on openstack_nova_server_status Prometheus metrics.

  • Updates Octavia images to fix Neutron endpoint selection in the OVN provider. LP#2049551.

  • Updates Octavia container images to fix a maintenance task that was breaking OVN IPv4 load balancers with health monitors. LP#2072754.

  • Pin the OCI image tag used for the Ubuntu Focal base-image of Kolla image builds. This prevents packages in the image with the latest tag getting in front of StackHPC release-train package repositories. Ubuntu tag should be bumped when new packages are available in StackHPC release-train.

  • Fixes an issue with Ansible Pulp modules depending on the pulp_glue Python library since the pulp.squeezer 0.0.14 release.

  • Pin the OCI image tag used for the base-image of Rocky 9 Kolla image builds. This prevents packages in the image with the latest tag getting in front of StackHPC release-train package repositories.

  • Fixes documentation builds on Read the Docs.

  • Fixes “Max Inlet Temp” time series chart in Redfish dashboard. This chart could wrongly display CPU2 temperature instead of inlet temperature.

  • Fixes a bug where the Redfish Dashboard’s UUID changed, causing Grafana to fail to update the dashboard on deployment where the old dashboard UUID had been used previously.

  • Changes the duration for which redfish exporter must continually fail scrapes before triggering an alert to 15 minutes. This should hopefully reduce some alert spam.

  • Removes bogus ContainerVolumeUsage alert. This rule wasn’t correctly measuring container volume IO and could cause spurious alerts.

  • Add a new reset-bls-entries.yml custom playbook which will rename existing Boot Loader Specification (BLS) entries using the current machine ID for each host. This should fix an issue with Grub not selecting the most recent kernel during boot.

  • Upstream package repository mirrors are now restored in Kolla container images. This makes it possible to install or update packages for debugging purposes.

  • Reverts “Track all interfaces in Keepalived” so only HA interfaces are tracked. This prevents L3 HA router flapping when detaching floating IP addresses, because non-HA router interfaces did not include “no_track”. Closes-Bug: #2097770

  • Fixed RADOS gateway usage exporter deployment failing to generate ec2 credentials for the ceph_rgw user.

  • Fixed RADOS gateway usage exporter not using the system trust root as its CA bundle.

  • Fixes synchronisation and DNF configuration of the Rocky Linux 9 CRB repository.

  • Fixes an issue with Kolla container image builds for Ubuntu where the release train package repositories could be behind the container image, leading to image build failures.

  • HAProxy alerting rules have been updated to use the server name that is down, rather than the name of the instance that reported the down server.

  • Fixes the issue with interface names containing dashes in Hashicorp collection.

  • Updates Magnum to a proper Caracal release version following critical bug upstream merged & backported. Also, updates CAPI Helm driver version to the latest (1.2.0).

  • The overcloud HashiCorp Vault playbooks have been modified to use the local Vault service rather than via HAProxy. This makes it possible to deploy and use Vault without HAProxy. This eliminates the previous bootstrapping issue where HAProxy needed to be deployed without TLS enabled while generating initial certificates.

Other Notes

  • deployment guide docs added for new capi driver

  • Reduced verbosity in etc/kayobe/pulp.yml

  • Changes the Grafana OpenStack dashboard to show HTTP status 300 as green instead of orange.

  • Adds a ci-aio environment for CI testing.

  • Adds a ci-builder environment for building Kolla container images in CI.