DA261 S3 backup end-to-end against MicroCeph#60
Open
delgod wants to merge 11 commits into
Open
Conversation
Default base implementation raises NotImplementedError so the protocol and the abstract base class remain importable and instantiable in between this commit and the substrate-specific overrides. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Stderr drained on a daemon thread to keep the kernel pipe buffer from blocking the streaming consumer. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Binary stdout (encoding=None), no harness timeout. Wraps ExecError into the (returncode, stderr) tuple convention. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reusable by BackupManager so the streaming RDB command shares the same TLS/auth construction as the rest of valkey-cli usage. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the BackupManager, BackupEvents, charm wiring, cross-cutting guards (storage_detaching, restart_workload), state model (ValkeyCluster.s3_credentials property, ValkeyServer.is_backup_in_progress, ClusterState.s3_relation, tls_paths.backup_ca), and the create-backup and list-backups actions. Backup runs on any unit (not only leader): the per-unit backup_id field in the peer unit databag serves as the lock and identifier. Streams valkey-cli --rdb - stdout directly to S3 via boto3 upload_fileobj. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Exercises the per-unit lock and any-unit guarantee by running backups from three different units and listing from a fourth. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an end-to-end integration test suite for the S3 backup feature, backed by a MicroCeph-based S3 fixture. Exercises the any-unit guarantee by running
create-backupon three different units andlist-backupson a fourth, then verifying the resulting objects in the bucket.Depends on
#59 must be merged first — this PR exercises actions and the relation surface introduced there.
What's in the PR
tests/integration/backup/__init__.py— package markertests/integration/backup/conftest.py:microcephfixture:snap install microceph, bootstrap,disk add loop,4G,3 --wipe, enable rgw with--ssl-port=445 --ssl-certificate=auto, create atestuser, generate a unique bucket name, expose the rgw self-signed certs3_bucketfixture: boto3Bucketresource (creates the bucket eagerly,verify=Falsefor the self-signed endpoint)s3_secret_content: access/secret-key dict for the Juju secret used bys3-integratortests/integration/backup/test_s3_backup.py:jubilants3-credsJuju secret, configures the integrator (bucket, endpoint, region, path, path-style URIs), waits foractive/idlecreate-backupon valkey/0 — asserts abackup-idmatching%Y-%m-%dT%H:%M:%SZcreate-backupon valkey/1 — asserts a distinct idlist-backupson valkey/2 — asserts both ids appear in the table in reverse-chronological orderREDISorVALKEY)Notes for reviewers
canonical/charmed-etcd-operator'stests/integration/backup/conftest.py(3.6/edge). The single test deviation from etcd-operator: a
--rdbmagic-byte verification of the uploaded object replaces etcd's "restore on a new model" round-trip, since restore is not yet implemented.