Add S3 support for non-Cinder backends#838
Open
himax16 wants to merge 2 commits into
Open
Conversation
himax16
force-pushed
the
feat/s3-support
branch
5 times, most recently
from
June 26, 2026 19:56
23283f8 to
41e3617
Compare
Signed-off-by: Himawan Winarto <himawan.winarto@canonical.com>
Signed-off-by: Himawan Winarto <himawan.winarto@canonical.com>
himax16
changed the title
himax16
marked this pull request as ready for review
June 30, 2026 15:01
Member
Author
Testing Guide with 2 LXD VMsArchitecturePrerequisites
Step 1: Create the LXD VMs# Create a shared network bridge if not already present
lxc network create lxdbr0 2>/dev/null || true
# VM1: MicroCeph node (16 GB RAM, 8 CPUs, 50 GB root + 80 GB OSD disk)
lxc init ubuntu:noble bm1 --vm -c limits.memory=16GB -c limits.cpu=8
lxc config device override bm1 root size=50GB
lxc storage volume create default bm1-osd0 --type block size=80GB
lxc storage volume create default bm1-osd1 --type block size=80GB
lxc storage volume create default bm1-osd2 --type block size=80GB
lxc config device add bm1 osd0 disk pool=default source=bm1-osd0
lxc config device add bm1 osd1 disk pool=default source=bm1-osd1
lxc config device add bm1 osd2 disk pool=default source=bm1-osd2
lxc network attach lxdbr0 bm1 eth0
lxc start bm1
# VM2: Sunbeam node (32 GB RAM, 8 CPUs, 100 GB root)
lxc init ubuntu:noble bm0 --vm -c limits.memory=32GB -c limits.cpu=8
lxc config device override bm0 root size=100GB
lxc network attach lxdbr0 bm0 eth0
lxc start bm0
# Wait for both VMs to boot
sleep 30Or alternatively, use the After the VMs are created, get their IP addresses: # Get IPs
VM1_IP=$(lxc exec bm1 -- hostname -I | awk '{print $1}')
VM2_IP=$(lxc exec bm0 -- hostname -I | awk '{print $1}')
echo "VM1 (microceph): $VM1_IP"
echo "VM2 (sunbeam): $VM2_IP"Step 2: Set Up MicroCeph on VM1# Enter VM1
lxc exec bm1 -- su - ubuntu
# Install microceph
sudo snap install microceph --channel=squid/stable
# Bootstrap the cluster
sudo microceph cluster bootstrap
# Add the OSD disk
## Find the disk (e.g., /dev/sdb) and add it
sudo microceph disk list
## Add each OSD disk (replace with actual disk IDs)
### When VMs are created manually
sudo microceph disk add --wipe /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_osd0
sudo microceph disk add --wipe /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_osd1
sudo microceph disk add --wipe /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_osd2
### When using `sunbeam-proxified-dev`
sudo microceph disk add --wipe /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_bm1_osd0
sudo microceph disk add --wipe /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_bm1_osd1
sudo microceph disk add --wipe /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_bm1_osd2
# Enable the Rados Gateway (S3)
sudo microceph enable rgw
# Verify services are running
sudo microceph status
# Expected: Services: mds, mgr, mon, rgw, osd
# Create an S3 user for Sunbeam
# NOTE: If it fails with "failed to fetch mon config", add --no-mon-config:
sudo microceph.radosgw-admin user create \
--uid=sunbeam-user \
--display-name="Sunbeam S3 User" \
--key-type=s3 \
--no-mon-config
# The output will contain access_key and secret_key
# Example output:
# {
# "user_id": "sunbeam-user",
# "display_name": "Sunbeam S3 User",
# "email": "",
# "suspended": 0,
# "max_buckets": 1000,
# "subusers": [],
# "keys": [
# {
# "user": "sunbeam-user",
# "access_key": "EPNVHHOUYSXMSY5LUFK1",
# "secret_key": "USRFVBGsVGwEaqvhwCPpV1UBeFz9nDySISh3K4wZ",
# "active": true,
# "create_date": "2026-06-26T12:19:37.961566Z"
# }
# ],
# "swift_keys": [],
# "caps": [],
# "op_mask": "read, write, delete",
# "default_placement": "",
# "default_storage_class": "",
# "placement_tags": [],
# "bucket_quota": {
# "enabled": false,
# "check_on_raw": false,
# "max_size": -1,
# "max_size_kb": 0,
# "max_objects": -1
# },
# "user_quota": {
# "enabled": false,
# "check_on_raw": false,
# "max_size": -1,
# "max_size_kb": 0,
# "max_objects": -1
# },
# "temp_url_keys": [],
# "type": "rgw",
# "mfa_ids": [],
# "account_id": "",
# "path": "/",
# "create_date": "2026-06-26T12:19:37.960731Z",
# "tags": [],
# "group_ids": []
# }
# Note the endpoint
echo "S3 Endpoint: http://$(hostname -I | awk '{print $1}'):80"
# Example:
# S3 Endpoint: http://192.167.98.11:80
# Exit VM1
exitStep 3: Verifying S3 AccessThroughout the rest of this guide, we assume these environment variables are already set on the host (e.g., from a export S3_ENDPOINT=http://192.167.98.11:80
export S3_ACCESS_KEY=EPNVHHOUYSXMSY5LUFK1
export S3_SECRET_KEY=USRFVBGsVGwEaqvhwCPpV1UBeFz9nDySISh3K4wZTo get these values from VM1 and persist them into VM2's # Fetch S3 endpoint and credentials from bm1
EP="http://$(lxc exec bm1 -- hostname -I | awk '{print $1}'):80"
AK="$(lxc exec bm1 -- microceph.radosgw-admin user info --uid=sunbeam-user --no-mon-config | jq -r '.keys[0].access_key')"
SK="$(lxc exec bm1 -- microceph.radosgw-admin user info --uid=sunbeam-user --no-mon-config | jq -r '.keys[0].secret_key')"
# Inject into VM2's .bashrc (persists across sessions)
lxc exec bm0 -- su - ubuntu -c "
echo 'export S3_ENDPOINT=$EP' >> /home/ubuntu/.bashrc
echo 'export S3_ACCESS_KEY=$AK' >> /home/ubuntu/.bashrc
echo 'export S3_SECRET_KEY=$SK' >> /home/ubuntu/.bashrc
"Go to the VM 2 shell and verify the variables are set: # Enter VM2
lxc exec bm0 -- su - ubuntu# Check if the S3 variables are set
echo "S3_ENDPOINT=$S3_ENDPOINT"
echo "S3_ACCESS_KEY=$S3_ACCESS_KEY"
echo "S3_SECRET_KEY=$S3_SECRET_KEY"Then install AWS CLI and verify S3 access from the host: snap install aws-cli --classic
aws configure set aws_access_key_id "$S3_ACCESS_KEY"
aws configure set aws_secret_access_key "$S3_SECRET_KEY"
aws configure set default.region us-east-1
aws s3 --endpoint-url "$S3_ENDPOINT" ls
aws s3 --endpoint-url "$S3_ENDPOINT" mb s3://test-bucket
echo 'Hello S3!' > ~/test.txt
aws s3 --endpoint-url "$S3_ENDPOINT" cp ~/test.txt s3://test-bucket/
aws s3 --endpoint-url "$S3_ENDPOINT" ls s3://test-bucket/
aws s3 --endpoint-url "$S3_ENDPOINT" cp s3://test-bucket/test.txt ~/downloaded.txt
diff ~/test.txt ~/downloaded.txt && echo "S3 WORKS!" || echo "FAILED"Step 4: Deploy s3-integrator on VM2# Copy the custom snap-openstack snap to VM2 (if testing from source)
lxc file push openstack_*_amd64.snap bm0/home/ubuntu/
# Enter VM2
lxc exec bm0 -- su - ubuntu
# S3 creds are already loaded from .bashrc (persisted in Step 3)
# Install Sunbeam snap from the local build
sudo snap install openstack_*_amd64.snap --dangerous
## Connect the juju connections
sudo snap connect openstack:juju-bin juju:juju-bin
sudo snap connect openstack:dot-local-share-juju
sudo snap connect openstack:dot-config-openstack
sudo snap connect openstack:dot-local-share-openstack
sudo snap connections openstack
# Prepare the nodes
sunbeam prepare-node-script --bootstrap | bash -x && newgrp snap_daemon
# Deploy the s3-integrator charm in the localhost-localhost controller
# Note: Any controller can be used, as long as Juju can reach it
# (This assumes Sunbeam is already bootstrapped)
# Create a model for the s3-integrator
juju add-model s3-storage
# Deploy s3-integrator from track 2 (uses Juju secrets for credentials)
juju deploy s3-integrator --channel=2/edge
# Configure the S3 connection (bucket auto-created by the charm)
juju config s3-integrator \
endpoint="$S3_ENDPOINT" \
region="us-east-1"
# Add a Juju secret containing the S3 access key and secret key
juju add-secret s3-creds-glance \
access-key="$S3_ACCESS_KEY" \
secret-key="$S3_SECRET_KEY"
# This will return a secret ID like:
# secret:auc58l8gqedugcc55ou0
# Grant the secret to s3-integrator
juju grant-secret s3-creds-glance s3-integrator
# Configure the charm with the secret reference
juju config s3-integrator credentials=secret:<secret-id>
# Wait for it to become active (charm verifies bucket is usable)
juju wait-for application s3-integrator --query='status=="active"'
# Create the cross-model offer
juju offer s3-integrator:s3-credentials
# Note the offer URL
juju offers
# Example output: admin/s3-storage.s3-integratorStep 5: Bootstrap and Deploy Sunbeam with S3Create a manifest file that uses the custom core:
...
software:
charms:
glance-k8s:
channel: 2026.1/edge/s3-support
s3-integrator-offer-url: "localhost-localhost:admin/s3-storage.s3-integrator"# Bootstrap Sunbeam cluster
sunbeam -v cluster bootstrap --role control,compute --manifest ~/manifest.yaml
# When there is a TF error check the logs in
# /home/ubuntu/snap/openstack/common/etc/<K8S-Cluster-Name>/terraform.log
# ... or by using
# sunbeam plans shell
# Verify glance is using S3
juju status glance
# Should show glance as active with s3-credentials relation
# Check glance config
juju show-unit glance/0
# Should show the S3 configurationStep 6: Verify Glance S3 Integration# Enter VM2 (su - loads .bashrc with persisted S3 creds)
# Load credentials
sunbeam openrc > ~/openrc
source ~/openrc
# Check that you are already authenticated with Keystone
openstack token issue
# Create a test image
echo "test image data" > ~/test.img
# Upload an image to Glance
openstack image create \
--file ~/test.img \
--disk-format raw \
--container-format bare \
test-s3-image
# List images
openstack image list
# Verify the image is stored in S3 (check the bucket)
aws s3 --endpoint-url "$S3_ENDPOINT" ls s3://glance/
# Should show the image data
# Delete the test image
openstack image delete test-s3-image
# Verify Glance cleaned up the backend storage properly
aws s3 --endpoint-url "$S3_ENDPOINT" ls s3://glance/Step 7: Test Telemetry with S3# Enter VM2 (su - loads .bashrc with persisted S3 creds)
lxc exec bm0 -- su - ubuntu
# Deploy a second s3-integrator for Gnocchi (separate instance per bucket, recommended)
## Switch to the localhost-localhost controller
juju switch localhost-localhost
juju deploy s3-integrator --channel=2/edge s3-integrator-gnocchi
# Configure with gnocchi bucket
juju config s3-integrator-gnocchi \
endpoint="$S3_ENDPOINT" \
region="us-east-1"
# There is an issue with "us-east-1" region
# Add and grant a Juju secret for gnocchi
juju add-secret s3-creds-gnocchi \
access-key="$S3_ACCESS_KEY" \
secret-key="$S3_SECRET_KEY"
# Returns a secret ID like: secret:abcdef1234567890abcd
juju grant-secret s3-creds-gnocchi s3-integrator-gnocchi
juju config s3-integrator-gnocchi credentials=secret:abcdef1234567890abcd
# Create cross-model offer for gnocchi
juju offer s3-integrator-gnocchi:s3-credentials
# Note the offer URL
juju offers
# Example output: admin/s3-storage.s3-integrator-gnocchi
# Enable telemetry with S3
sunbeam enable telemetry --s3-integrator-offer-url "localhost-localhost:admin/s3-storage.s3-integrator"
# Verify gnocchi is using S3
juju status gnocchi
juju show-unit gnocchi/0
# Should show driver = s3
# Log into the OpenStack as admin
source ~/openrc
openstack token issue
# Create a test metric
openstack metric resource create --type generic test-resource
openstack metric create test-metric --resource-id test-resource
openstack metric measures add test-metric --resource-id test-resource -m $(date -u +%Y-%m-%dT%H:%M:%SZ)@42
# Check the Gnocchi metrics resource list
openstack metric resource list
# Confirm that metrics are being stored in S3 (check the bucket)
# The configuration ``bucket:gnocchi`` is treated as a prefix for the actual
# S3 buckets used.
aws s3 --endpoint-url "$S3_ENDPOINT" ls s3://gnocchi-aggregates/
aws s3 --endpoint-url "$S3_ENDPOINT" ls s3://gnocchi-measure/Step 8: Test Ironic with S3# Enter VM2 (su - loads .bashrc with persisted S3 creds)
lxc exec bm0 -- su - ubuntu
# Enable baremetal feature
sunbeam enable baremetal
# Verify ironic-conductor has the s3-credentials relation
juju status ironic-conductor
# Should show s3-credentials relation
# Verify temp-url-secret is NOT required
juju ssh ironic-conductor/0 -- cat /etc/ironic/ironic.conf | grep -E 'swift_temp_url|image_download|ipxe_use_swift'
# Should show:
# image_download_source = local
# (no swift_temp_url_key)
# (no ipxe_use_swift)Step 9: Cleanup# Destroy VMs
lxc stop bm1 bm0
lxc delete bm1 bm0
# Remove the network bridge (optional)
lxc network delete lxdbr0 |
|
This PR is predicated on the spec, thus it also needs reviewed and a consensus on the approach before this can land. |
hmlanigan
reviewed
Jul 6, 2026
| if is_glance_s3_storage_enabled(self.deployment.get_client()): | ||
| return Result( | ||
| ResultType.SKIPPED, | ||
| "Glance uses external S3 storage; temp-url secret not required.", |
There was a problem hiding this comment.
Suggested change
| "Glance uses external S3 storage; temp-url secret not required.", | |
| "Glance is using external S3 storage; set-temp-url-secret action not run.", |
| self.apps = apps | ||
|
|
||
| def is_skip(self, context: StepContext) -> Result: | ||
| """Skip the temp-url-secret action when Glance uses external S3. |
There was a problem hiding this comment.
s/temp-url-secret/set-temp-url-secret/
set-temp-url-secret is the action per the ironic charmcraft.yaml
| LOG = logging.getLogger(__name__) | ||
| console = Console() | ||
|
|
||
| TELEMETRY_METRICS_BACKEND_KEY = "TelemetryMetricsBackend" |
There was a problem hiding this comment.
Suggested change
| TELEMETRY_METRICS_BACKEND_KEY = "TelemetryMetricsBackend" | |
| TELEMETRY_BACKEND_KEY = "TelemetryBackend" |
Metrics is redundant for this key and content name.
Comment on lines
+48
to
+49
| class MetricsBackendType(enum.Enum): | ||
| """Telemetry metrics storage backend types.""" |
There was a problem hiding this comment.
todo: please consistently use the term telemetry rather than mixing in metrics in some places too. It's a bit confusing.
| # Telemetry service | ||
|
|
||
| This feature provides Telemetry service for Sunbeam. It is based on OpenStack Telemetry projects [Ceilometer](https://docs.openstack.org/designate/latest/), [Aodh](https://docs.openstack.org/aodh/latest/), [Gnocchi](https://wiki.openstack.org/wiki/Gnocchi). | ||
| This feature provides Telemetry service for Sunbeam. It is based on OpenStack Telemetry projects [Ceilometer](https://docs.openstack.org/ceilometer/latest/), [Aodh](https://docs.openstack.org/aodh/latest/), [Gnocchi](https://wiki.openstack.org/wiki/Gnocchi). |
There was a problem hiding this comment.
Suggested change
| This feature provides Telemetry service for Sunbeam. It is based on OpenStack Telemetry projects [Ceilometer](https://docs.openstack.org/ceilometer/latest/), [Aodh](https://docs.openstack.org/aodh/latest/), [Gnocchi](https://wiki.openstack.org/wiki/Gnocchi). | |
| This feature provides a Telemetry service for Sunbeam. It is based on OpenStack Telemetry projects: [Ceilometer](https://docs.openstack.org/ceilometer/latest/), [Aodh](https://docs.openstack.org/aodh/latest/), and [Gnocchi](https://wiki.openstack.org/wiki/Gnocchi). |
| - Gnocchi: Time series database service [charm](https://opendev.org/openstack/charm-gnocchi-k8s) [ROCK](https://github.com/canonical/ubuntu-openstack-rocks/tree/main/rocks/gnocchi-consolidated) | ||
| - Ceilometer Agent: Agent on hypervisor [charm](https://opendev.org/openstack/charm-openstack-hypervisor) [SNAP](https://github.com/canonical/snap-openstack-hypervisor.git) | ||
| - MySQL Router for Designate [charm](https://github.com/canonical/mysql-router-k8s-operator) [ROCK](https://github.com/canonical/charmed-mysql-rock) | ||
| - TODO: ADD OPENSTACK EXPORTER INFO |
|
|
||
|
|
||
| def is_glance_s3_storage_enabled(client: Client) -> bool: | ||
| """Whether the deployed control plane uses external S3 for Glance images.""" |
There was a problem hiding this comment.
Suggested change
| """Whether the deployed control plane uses external S3 for Glance images.""" | |
| """Whether the deployed control plane uses S3 for Glance images.""" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add S3 support for non-Cinder backends utilizing integration with the
s3-connectorcharm (track 2).Related PR:
gnocchi-k8sgnocchi-k8sfixglance-k8sandironic-conductor-k8sOPEN-4541