Skip to content

DA261 S3 backup end-to-end against MicroCeph#60

Open
delgod wants to merge 11 commits into
9/edgefrom
s3-backup-integration-tests
Open

DA261 S3 backup end-to-end against MicroCeph#60
delgod wants to merge 11 commits into
9/edgefrom
s3-backup-integration-tests

Conversation

@delgod
Copy link
Copy Markdown
Member

@delgod delgod commented May 13, 2026

Summary

Adds an end-to-end integration test suite for the S3 backup feature, backed by a MicroCeph-based S3 fixture. Exercises the any-unit guarantee by running create-backup on three different units and list-backups on a fourth, then verifying the resulting objects in the bucket.

Depends on

#59 must be merged first — this PR exercises actions and the relation surface introduced there.

What's in the PR

  • tests/integration/backup/__init__.py — package marker
  • tests/integration/backup/conftest.py:
    • Module-scoped microceph fixture: snap install microceph, bootstrap, disk add loop,4G,3 --wipe, enable rgw with --ssl-port=445 --ssl-certificate=auto, create a test user, generate a unique bucket name, expose the rgw self-signed cert
    • s3_bucket fixture: boto3 Bucket resource (creates the bucket eagerly, verify=False for the self-signed endpoint)
    • s3_secret_content: access/secret-key dict for the Juju secret used by s3-integrator
  • tests/integration/backup/test_s3_backup.py:
    • Deploys 3 valkey units + s3-integrator on Ubuntu 24.04 via jubilant
    • Provisions an s3-creds Juju secret, configures the integrator (bucket, endpoint, region, path, path-style URIs), waits for active/idle
    • Integrates valkey ↔ s3-integrator
    • Runs create-backup on valkey/0 — asserts a backup-id matching %Y-%m-%dT%H:%M:%SZ
    • Runs create-backup on valkey/1 — asserts a distinct id
    • Runs list-backups on valkey/2 — asserts both ids appear in the table in reverse-chronological order
    • Verifies via boto3 that two objects exist under the configured prefix and the first object's body starts with the RDB magic bytes (REDIS or VALKEY)

Notes for reviewers

  • The MicroCeph fixture is adapted from canonical/charmed-etcd-operator's tests/integration/backup/conftest.py
    (3.6/edge). The single test deviation from etcd-operator: a --rdb magic-byte verification of the uploaded object replaces etcd's "restore on a new model" round-trip, since restore is not yet implemented.

delgod and others added 11 commits May 13, 2026 13:23
Default base implementation raises NotImplementedError so the protocol
and the abstract base class remain importable and instantiable in
between this commit and the substrate-specific overrides.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Stderr drained on a daemon thread to keep the kernel pipe buffer from
blocking the streaming consumer.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Binary stdout (encoding=None), no harness timeout. Wraps ExecError into
the (returncode, stderr) tuple convention.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reusable by BackupManager so the streaming RDB command shares the same
TLS/auth construction as the rest of valkey-cli usage.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the BackupManager, BackupEvents, charm wiring, cross-cutting
guards (storage_detaching, restart_workload), state model
(ValkeyCluster.s3_credentials property, ValkeyServer.is_backup_in_progress,
ClusterState.s3_relation, tls_paths.backup_ca), and the create-backup
and list-backups actions.

Backup runs on any unit (not only leader): the per-unit backup_id field
in the peer unit databag serves as the lock and identifier. Streams
valkey-cli --rdb - stdout directly to S3 via boto3 upload_fileobj.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Exercises the per-unit lock and any-unit guarantee by running backups
from three different units and listing from a fourth.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant