Skip to content

feat: add support for RGW in ci-multinode #2337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
jackhodgkiss wants to merge 1 commit into
base: stackhpc/2025.1
Choose a base branch
from
Draft

feat: add support for RGW in ci-multinode #2337

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 48 additions & 1 deletion etc/kayobe/environments/ci-multinode/cephadm.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,54 @@ cephadm_keys:
# List of Cephadm commands to run. See stackhpc.cephadm.commands role for format.
cephadm_commands_pre: []

cephadm_commands_post: "{{ cephadm_commands_manila_cephfs_native if (kolla_enable_manila | bool and kolla_enable_manila_backend_cephfs_native | bool) else [] }}"
cephadm_commands_post: >-
{{
(cephadm_commands_manila_cephfs_native
if (kolla_enable_manila | bool and kolla_enable_manila_backend_cephfs_native | bool)
else [])
+
(cephadm_commands_rgw
if (kolla_enable_ceph_rgw | bool)
else [])
}}
cephadm_commands_manila_cephfs_native:
- "fs new manila-cephfs cephfs_metadata cephfs_data"
- "orch apply mds manila-cephfs"

cephadm_commands_rgw:
- "config set client.rgw rgw_content_length_compat true"
- "config set client.rgw rgw_enable_apis 's3, swift, swift_auth, admin'"
- "config set client.rgw rgw_enforce_swift_acls true"
- "config set client.rgw rgw_keystone_accepted_admin_roles 'admin'"
- "config set client.rgw rgw_keystone_accepted_roles 'member, admin'"
- "config set client.rgw rgw_keystone_admin_domain Default"
- "config set client.rgw rgw_keystone_admin_password {{ (lookup('file', kayobe_env_config_path ~ '/kolla/passwords.yml') | from_yaml).ceph_rgw_keystone_password }}"
- "config set client.rgw rgw_keystone_admin_project service"
- "config set client.rgw rgw_keystone_admin_user 'ceph_rgw'"
- "config set client.rgw rgw_keystone_api_version '3'"
- "config set client.rgw rgw_keystone_token_cache_size '10000'"
- "config set client.rgw rgw_keystone_url {{ 'https' if kolla_enable_tls_internal | bool else 'http' }}://{{ kolla_internal_fqdn }}:5000"
- "config set client.rgw rgw_keystone_verify_ssl true"
- "config set client.rgw rgw_max_attr_name_len '1000'"
- "config set client.rgw rgw_max_attr_size '1000'"
- "config set client.rgw rgw_max_attrs_num_in_req '1000'"
- "config set client.rgw rgw_s3_auth_use_keystone true"
- "config set client.rgw rgw_swift_account_in_url true"
- "config set client.rgw rgw_swift_versioning_enabled true"

cephadm_radosgw_services:
- id: myrgw
count_per_host: 1
spec:
rgw_frontend_port: 8100

ca_bundle_path: >-
{{
'/etc/ssl/certs/ca-certificates.crt'
if os_distribution == 'ubuntu'
else '/etc/pki/tls/certs/ca-bundle.crt'
}}
Comment on lines +115 to +120

@gemini-code-assist gemini-code-assist Bot Jun 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The variable os_release is not standard in this environment's configuration files. Other files (such as kolla.yml, storage.yml, and kolla/globals.yml) consistently use os_distribution to determine the OS flavor (e.g., 'ubuntu'). Using os_release here may result in an undefined variable error or incorrect evaluation.

ca_bundle_path: >-
  {{
    '/etc/ssl/certs/ca-certificates.crt'
    if os_distribution == 'ubuntu'
    else '/etc/pki/tls/certs/ca-bundle.crt'
  }}


cephadm_extra_container_args:
- "-v"
- "{{ ca_bundle_path }}:/etc/pki/tls/certs/ca-bundle.crt:ro"
2 changes: 2 additions & 0 deletions etc/kayobe/environments/ci-multinode/inventory/group_vars/storage/network-interfaces
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ admin_oc_interface: "{{ ansible_facts.default_ipv4.interface }}"

internal_interface: "{{ vxlan_interfaces[0].device }}.{{ internal_vlan }}"

public_interface: "{{ vxlan_interfaces[0].device }}.{{ public_vlan }}"

storage_interface: "{{ vxlan_interfaces[0].device }}.{{ storage_vlan }}"

storage_mgmt_interface: "{{ vxlan_interfaces[0].device }}.{{ storage_mgmt_vlan }}"
Expand Down
1 change: 1 addition & 0 deletions etc/kayobe/environments/ci-multinode/kolla.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
---
kolla_enable_ceph_rgw: true
kolla_enable_cinder: true
kolla_enable_cinder_backup: true
kolla_enable_neutron_provider_networks: true
Expand Down
15 changes: 15 additions & 0 deletions etc/kayobe/environments/ci-multinode/kolla/globals.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,18 @@ external_api_firewalld_zone: "{{ public_net_name | net_zone }}"

# Test cinder cluster
cinder_cluster_name: "stackhpc_test_cinder_cluster"

# RGW configuration
ceph_rgw_swift_compatibility: false
ceph_rgw_swift_account_in_url: true

ceph_rgw_hosts:
- host: "{{ hostvars[groups['rgws'][0]].ansible_hostname }}"
ip: "{{ lookup('vars', storage_net_name ~ '_ips')[groups.rgws.0] }}"
port: 8100
- host: "{{ hostvars[groups['rgws'][1]].ansible_hostname }}"
ip: "{{ lookup('vars', storage_net_name ~ '_ips')[groups.rgws.1] }}"
port: 8100
- host: "{{ {{ hostvars[groups['rgws'][2]].ansible_hostname }} }}"
ip: "{{ lookup('vars', storage_net_name ~ '_ips')[groups.rgws.2] }}"
port: 8100
Comment on lines +78 to +87

@gemini-code-assist gemini-code-assist Bot Jun 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using groups.rgws is problematic for two reasons:

  1. The standard Kolla Ansible group name for Ceph RGW is rgw, not rgws, which would cause a template rendering failure.
  2. Relying on the list index of the group (e.g., groups.rgws.0) is fragile because it assumes a specific ordering of hosts in the inventory and that there are at least 3 hosts in that group.

Since the hostnames (storage-01, storage-02, storage-03) are already hardcoded here, it is much safer and more robust to look up their IPs directly by hostname from the <network>_ips dictionary.

ceph_rgw_hosts:
  - host: storage-01
    ip: "{{ lookup('vars', storage_net_name ~ '_ips')['storage-01'] }}"
    port: 8100
  - host: storage-02
    ip: "{{ lookup('vars', storage_net_name ~ '_ips')['storage-02'] }}"
    port: 8100
  - host: storage-03
    ip: "{{ lookup('vars', storage_net_name ~ '_ips')['storage-03'] }}"
    port: 8100

19 changes: 18 additions & 1 deletion etc/kayobe/environments/ci-multinode/storage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,23 @@
---
###############################################################################
# Storage node configuration.

# User with which to access the storage nodes via SSH during bootstrap, in
# order to setup the Kayobe user account. Default is 'cloud-user' if
# os_distribution is set to centos, otherwise 'os_distribution'.
storage_bootstrap_user: "{{ os_distribution if os_distribution == 'ubuntu' else 'cloud-user' }}"
# List of storage volume groups. See mrlesmithjr.manage-lvm role for

###############################################################################
# Storage network interface configuration.

# List of extra networks to which storage nodes are attached.
storage_extra_network_interfaces:
- public

###############################################################################
# Storage node LVM configuration.

# List of storage volume groups. See mrlesmithjr.manage_lvm role for
# format.
storage_lvm_groups:
- "{{ stackhpc_lvm_group_rootvg }}"
Expand Down
Loading