Build Capsule's offline end-to-end encrypted data plane

DEFERRED.md

@@ -0,0 +1,78 @@
+# Deferred Work
+
+This file records what the **core data-plane implementation** (the `capsule-core`
+cryptographic core + offline lifecycle + `capsule demo`) intentionally left for later, why,
+and the seam that was left in place so it can drop in without reworking what exists. It
+complements the design docs in `capsule-docs/src/content/docs/design/`.
+
+## What is implemented and validated (offline, real crypto)
+
+`capsule-core` implements and exhaustively unit-tests the full offline data plane:
+
+- **Canonical CBOR** (RFC 8949 §4.2) — the byte-identity contract for every signature/hash.
+- **Crypto primitives** — SHA-256 (streaming), HKDF-SHA512, Argon2id, AES-256-GCM (STREAM +
+  standalone metadata-blob), **hybrid Ed25519 + ML-DSA-65** signatures (both halves required),
+  **ML-KEM-768** DEK.
+- **Key hierarchy** — master key, default-album-id derivation, AMKs + per-file/blob keys,
+  software keystore (account ↔ encrypted `AccountFile`), signed device directory.
+- **Encryption** — STREAM asset encryption with independent ranged-chunk decryption;
+  exact metadata-blob wire format.
+- **Provenance** — signed `AssetManifest`/`DerivativeManifest`, append-only hash-chained
+  provenance, and the single **`verify_asset`** chokepoint (Accept / TerminalReject / Pending)
+  with an exhaustive negative-case suite.
+- **Validation invariants** — the key-less protocol handshake + structural envelope checks +
+  idempotency keys.
+- **CRDT metadata + Sidecar v1** — OR-set tags, LWW caption/rating with superseded log,
+  monotonic add-id counter; signed `SidecarV1` (schema as CBOR field 0); privacy-on-export.
+- **Backup** — deterministic signed tar artifact, AMK ledger, master-key escrow, Shamir
+  2-of-3, and dry-run/commit restore with chain reconciliation.
+- **Lifecycle `Workspace`** — ties it together and is showcased end-to-end by `capsule demo`.
+
+## Deferred — with the seam in place
+
+### Real MLS / OpenMLS group state
+- **Why:** the design's MLS ciphersuite (`MLS_256_XWING_CHACHA20POLY1305_SHA256_Ed25519`,
+  `0x004D`) exists in `openmls` only via a C (`libcrux`) backend on a non-final IETF draft,
+  with no IANA codepoint and no RustCrypto PQ backend yet (openmls#1940).
+- **Seam:** `capsule_core::crypto::authority::AlbumAuthority` is the trait `verify_asset`
+  consumes (epoch ceiling, per-epoch write-tier pubkey, AMK presence, admin-chain validity).
+  `ReferenceAuthority` (an admin-signed epoch ledger) stands in for live MLS and is honored
+  only via `&dyn AlbumAuthority`, so an `OpenMlsAuthority` drops in unchanged.
+- **Consequence:** albums are **single-epoch** in the offline core. Epoch rotation,
+  membership add/remove, the `Welcome`/history-delivery flow, and the album upgrade ceremony
+  are deferred with OpenMLS.
+
+### X-Wing hybrid DEK
+- `crypto::keys::kem` implements the post-quantum **ML-KEM-768** half (full encapsulate/
+  decapsulate round-trip). The X25519 classical half and the X-Wing combiner land with
+  OpenMLS (the seam is byte-string `encapsulate`/`decapsulate`, combiner-agnostic).
+
+### Hardware-bound key storage
+- Device keys are kept in a **software keystore** (private keys sealed under the
+  passphrase-wrapped master key). Secure Enclave / StrongBox / TPM adapters
+  (`capsule-sdk::hardware-keys`) are per-platform glue, deferred.
+
+### Networked server/client
+- All transport is out of scope here: the HTTP/TUS upload server, GraphQL resolvers, the
+  `/sync` feed, federation, peering, and the `capsule-sdk` network client. The **pure**
+  refuse-by-default validation invariants those paths need are implemented in
+  `capsule_core::validation` and ready to wire into `capsule-api`.
+
+### ML / AI
+- Embeddings, `sqlite-vec` vector search, the model registry, semantic/face features, and
+  moderation are deferred (explicitly out of scope). The sidecar reserves `tags_ai`
+  (separate OR-set) and the manifest reserves `model_id`/`model_version` for them.
+
+### Other
+- Thumbnail/LQIP generation beyond `capsule-media`'s existing utilities.
+- Fusing the crypto data plane into the **existing plaintext import executor**
+  (`capsule_core::import::executor`): that pipeline still writes the legacy `AssetSidecar`.
+  The crypto-integrated lifecycle lives in `capsule_core::lifecycle::Workspace` (used by
+  `capsule demo`); unifying the two import paths is a follow-up.
+
+## How to see it working
+
+```
+cargo test --workspace --exclude capsule-sdk      # full unit + e2e test surface
+cargo run -p capsule-cli -- demo --workdir /tmp/capsule-demo   # narrated end-to-end showcase
+```

capsule-cli/src/cli/commands.rs

@@ -28,6 +28,15 @@ pub enum Commands {
         #[command(subcommand)]
         command: LibraryCommands,
     },
+    /// Run the offline end-to-end data-plane showcase (real cryptography, no network)
+    Demo {
+        /// Working directory for the demo libraries (a temp dir is used if omitted)
+        #[arg(long, value_name = "PATH")]
+        workdir: Option<PathBuf>,
+        /// A real image/file to import (a small synthetic file is used if omitted)
+        #[arg(long, value_name = "PATH")]
+        image: Option<PathBuf>,
+    },
     /// Sync local and remote data
     Sync {
         /// Force sync even if there are conflicts

capsule-cli/src/demo.rs

@@ -0,0 +1,221 @@
+//! `capsule demo` — an offline, end-to-end showcase of the cryptographic data plane.
+//!
+//! Runs the whole flow with **real cryptography and no network**: account + keys → album +
+//! authority → import (encrypt, signed manifest, provenance, signed sidecar, `verify_asset`)
+//! → CRDT metadata edits → soft-delete + restore → backup export → restore into a fresh
+//! library → byte-equal verification → Shamir 2-of-3 recovery. Every step writes real
+//! artifacts the user can inspect.
+
+use std::path::PathBuf;
+
+use colored::*;
+use eyre::{Result, eyre};
+
+use capsule_core::backup::{recover_seed, split_seed_2of3};
+use capsule_core::crypto::primitives::Argon2Params;
+use capsule_core::crypto::verify_asset::VerifyOutcome;
+use capsule_core::lifecycle::Workspace;
+
+/// Fast Argon2id parameters — this is a demo; the wrap-key strength is not the point.
+const DEMO_KDF: Argon2Params = Argon2Params {
+    mem_kib: 8 * 1024,
+    t_cost: 1,
+    p_cost: 1,
+};
+
+fn step(n: u32, title: &str) {
+    println!("\n{} {}", format!("[{n}]").bold().cyan(), title.bold());
+}
+
+fn ok(msg: impl AsRef<str>) {
+    println!("    {} {}", "✓".green(), msg.as_ref());
+}
+
+fn info(label: &str, value: impl std::fmt::Display) {
+    println!("    {} {}", format!("{label}:").dimmed(), value);
+}
+
+/// Run the showcase. `workdir` defaults to a fresh temp directory; `image` defaults to a
+/// small synthetic file.
+pub fn run(workdir: Option<PathBuf>, image: Option<PathBuf>) -> Result<()> {
+    let root = match workdir {
+        Some(p) => p,
+        None => std::env::temp_dir().join(format!("capsule-demo-{}", std::process::id())),
+    };
+    std::fs::create_dir_all(&root)?;
+    let source_lib = root.join("source-library");
+    let fresh_lib = root.join("restored-library");
+    let backup_path = root.join("backup.tar");
+
+    println!(
+        "{}",
+        "Capsule offline data-plane showcase (real crypto, no network)"
+            .bold()
+            .underline()
+    );
+    info("workdir", root.display());
+
+    // ── 1. Account + device keys ────────────────────────────────────────────
+    step(1, "Create account + device keys");
+    let mut ws = Workspace::create_with_params(&source_lib, b"demo-passphrase", DEMO_KDF)
+        .map_err(|e| eyre!("create workspace: {e}"))?;
+    ok(
+        "master key generated; identity (IK), device signing (DSK), and device encryption (DEK) keys created",
+    );
+    info("user_id", ws.user_id());
+    info(
+        "default album id (derived from master key)",
+        ws.default_album_id(),
+    );
+
+    // ── 2. Album + MLS-attested authority ───────────────────────────────────
+    step(
+        2,
+        "Create a container album (mint AMK + write-tier + admin keys)",
+    );
+    let album = ws.create_album("Trip to the Coast");
+    ok("AMK_v1 minted; admin-signed authority attests epoch 1");
+    info("album_id", album);
+
+    // ── 3. Import a real file ────────────────────────────────────────────────
+    step(
+        3,
+        "Import an asset (encrypt → sign manifest → provenance → signed sidecar → verify_asset)",
+    );
+    let image_path = match image {
+        Some(p) => p,
+        None => {
+            let p = root.join("sample.jpg");
+            // A small synthetic JPEG-ish payload.
+            let mut bytes = vec![0xFF, 0xD8, 0xFF, 0xE0];
+            bytes.extend((0..4096).map(|i| (i % 256) as u8));
+            std::fs::write(&p, &bytes)?;
+            p
+        }
+    };
+    info("source file", image_path.display());
+    let asset = ws
+        .import_asset(album, &image_path)
+        .map_err(|e| eyre!("import: {e}"))?;
+    let st = ws.asset(&asset).ok_or_else(|| eyre!("asset missing"))?;
+    let head = &st.chain.records().last().unwrap().manifest;
+    ok("encrypted with AES-256-GCM STREAM; manifest signed (device + write-tier hybrid sigs)");
+    info("asset_id", asset);
+    info("ciphertext hash", head.core.ciphertext_hash);
+    info(
+        "plaintext size",
+        format!("{} bytes", head.core.plaintext_size),
+    );
+    ok("signed sidecar + provenance chain written to disk under media/");
+
+    // ── 4. verify_asset chokepoint ───────────────────────────────────────────
+    step(4, "Acknowledge via the verify_asset chokepoint");
+    match ws.verify(&asset).map_err(|e| eyre!("verify: {e}"))? {
+        VerifyOutcome::Accept => {
+            ok("verify_asset → ACCEPT (both signatures, epoch, chain, AMK all valid)")
+        }
+        other => return Err(eyre!("unexpected verify outcome: {other:?}")),
+    }
+
+    // ── 5. CRDT metadata edits ───────────────────────────────────────────────
+    step(5, "Collaborative metadata edits (CRDT, provenance-tracked)");
+    ws.tag_add(&asset, "coast").map_err(|e| eyre!("tag: {e}"))?;
+    ws.tag_add(&asset, "sunset")
+        .map_err(|e| eyre!("tag: {e}"))?;
+    ws.set_caption(&asset, "golden hour over the bay")
+        .map_err(|e| eyre!("caption: {e}"))?;
+    let st = ws.asset(&asset).unwrap();
+    let tags: Vec<String> = st.sidecar.tags_user.value().into_iter().collect();
+    ok(format!("tags (OR-set): {tags:?}"));
+    ok(format!(
+        "caption (LWW): {:?}",
+        st.sidecar.caption.get().cloned().unwrap_or_default()
+    ));
+    info("provenance records", st.chain.records().len());
+
+    // ── 6. Lifecycle: soft delete + restore ──────────────────────────────────
+    step(6, "Soft-delete (signed retention window) then restore");
+    ws.soft_delete(&asset, 30)
+        .map_err(|e| eyre!("delete: {e}"))?;
+    ok("delete manifest signed with retention_until = now + 30 days");
+    ws.restore(&asset).map_err(|e| eyre!("restore: {e}"))?;
+    ok("trash-restore appended; the delete record is preserved in the chain (audit trail)");
+    let st = ws.asset(&asset).unwrap();
+    let actions: Vec<String> = st
+        .chain
+        .records()
+        .iter()
+        .map(|r| format!("{:?}", r.manifest.core.action).to_lowercase())
+        .collect();
+    info("chain", actions.join(" → "));
+
+    // ── 7. Backup export ─────────────────────────────────────────────────────
+    step(7, "Export a portable, signed backup artifact");
+    ws.export_backup(&backup_path, b"recovery-passphrase")
+        .map_err(|e| eyre!("export: {e}"))?;
+    let size = std::fs::metadata(&backup_path)?.len();
+    ok("deterministic tar with HMAC + hybrid-signed MANIFEST + sealed AMK ledger");
+    info(
+        "backup",
+        format!("{} ({} bytes)", backup_path.display(), size),
+    );
+
+    // ── 8. Restore into a fresh library ──────────────────────────────────────
+    step(8, "Restore into a FRESH library and verify byte-equality");
+    let exporter_pub = ws.exporter_verifying_key();
+    let mut fresh = Workspace::create_with_params(&fresh_lib, b"new-device-pass", DEMO_KDF)
+        .map_err(|e| eyre!("create fresh: {e}"))?;
+    let added = fresh
+        .import_backup(&backup_path, b"recovery-passphrase", &exporter_pub)
+        .map_err(|e| eyre!("import backup: {e}"))?;
+    ok(format!(
+        "restored {added} asset(s) after verifying the exporter signature + AMK completeness"
+    ));
+    let original = ws
+        .read_plaintext(&asset)
+        .map_err(|e| eyre!("read src: {e}"))?;
+    let restored = fresh
+        .read_plaintext(&asset)
+        .map_err(|e| eyre!("read restored: {e}"))?;
+    if original == restored {
+        ok(format!(
+            "{} restored plaintext is byte-identical to the source",
+            "PASS:".green().bold()
+        ));
+    } else {
+        return Err(eyre!("restored plaintext differs from source"));
+    }
+
+    // A wrong exporter key (untrusted device) is refused.
+    let bogus = Workspace::create_with_params(&root.join("bogus"), b"x", DEMO_KDF)
+        .map_err(|e| eyre!("bogus ws: {e}"))?
+        .exporter_verifying_key();
+    let mut reject_lib = Workspace::create_with_params(&root.join("reject"), b"x", DEMO_KDF)
+        .map_err(|e| eyre!("reject ws: {e}"))?;
+    match reject_lib.import_backup(&backup_path, b"recovery-passphrase", &bogus) {
+        Err(_) => ok("a backup signed by an untrusted exporter is refused"),
+        Ok(_) => return Err(eyre!("untrusted exporter backup was wrongly accepted")),
+    }
+
+    // ── 9. Shamir social recovery ────────────────────────────────────────────
+    step(9, "Opt-in Shamir 2-of-3 recovery-seed sharing");
+    let seed = [0x5Au8; 32];
+    let shares = split_seed_2of3(&seed);
+    let recovered =
+        recover_seed(&[shares[0].clone(), shares[2].clone()]).map_err(|e| eyre!("shamir: {e}"))?;
+    if recovered == seed {
+        ok("split into 3 shares; any 2 reconstruct the seed (1 alone reveals nothing)");
+    } else {
+        return Err(eyre!("shamir reconstruction failed"));
+    }
+
+    println!(
+        "\n{}  Every layer of the design exercised offline with real cryptography.",
+        "DONE.".green().bold()
+    );
+    println!(
+        "{}",
+        format!("Inspect the on-disk artifacts under {}", root.display()).dimmed()
+    );
+    Ok(())
+}

capsule-cli/src/import/plan.rs

@@ -2,9 +2,9 @@
 
 use std::path::PathBuf;
 
-use eyre::{Result, eyre};
 use capsule_core::import::ImportActionPlan;
 use capsule_core::import::scanner::scan;
+use eyre::{Result, eyre};
 
 /// Scan a file or directory and build an import action plan.
 ///

capsule-cli/src/main.rs

@@ -1,18 +1,18 @@
 use std::path::Path;
 
 use capitalize::Capitalize;
-use clap::Parser;
-use cli::{AuthCommands, Cli, Commands, LibraryCommands};
-use colored::*;
-use dialoguer::Confirm;
-use eyre::{Result, eyre};
 use capsule_core::domain::ImportMode;
 use capsule_core::import::scanner::scan as scan_files;
 use capsule_core::import::{
     CancellationToken, ImportConfig, ImportOutcome, ImportProgressEvent, execute, plan,
 };
 use capsule_core::library::{Library, LibraryError, init_library, open_library, rebuild_index};
 use capsule_core::metadata::FileMetadata;
+use clap::Parser;
+use cli::{AuthCommands, Cli, Commands, LibraryCommands};
+use colored::*;
+use dialoguer::Confirm;
+use eyre::{Result, eyre};
 use tracing::trace;
 use tracing_subscriber::prelude::*;
 use tracing_subscriber::{EnvFilter, fmt};
@@ -22,6 +22,7 @@ use crate::utils::directories::{get_cache_dir, get_config_dir, get_data_dir};
 mod cli;
 mod config;
 mod db;
+mod demo;
 mod import;
 mod status;
 mod utils;
@@ -237,6 +238,11 @@ async fn main() -> Result<()> {
                 .map_err(|e| eyre!("Failed to close library: {e}"))?;
         }
 
+        // ── Demo ──────────────────────────────────────────────────────────
+        Commands::Demo { workdir, image } => {
+            demo::run(workdir, image)?;
+        }
+
         // ── Sync ──────────────────────────────────────────────────────────
         Commands::Sync { force, dry_run } => {
             println!("{}", "Syncing local and remote data...".green());