Store backends (SQLite + PostgreSQL) and remote storage (MinIO first): first integration phase #506

Activity

1. yifanxuaaa commented on Oct 29, 2026

   

   yifanxuaaa

   on Oct 29, 2026

   ContributorAuthor

   More actions

   Updated the description with Design adjustments (home-level remotes, push and pull state outside the Store, where each project's Store is recorded, pull exclusivity, snapshot staging space, worker networking, no HTTP Store server) and a new part c. API layer and Connect wiring. Session b was sent items 1–6 directly. Progress so far: 514a79f (snapshot) and a9903e1 (SQLite code grouped under backend/sqlite).
2. yifanxuaaa commented on Oct 29, 2026

   

   yifanxuaaa

   on Oct 29, 2026

   ContributorAuthor

   More actions

   Area R (remote storage, MinIO first): done for this phase — session b

   Draft PR #507 (shared with area A). Commits 5586d8f6af, b55e317c6e, 9c6ebef636. Description: core/docs/architecture/86-remote-store-files.md.

   What was built

   • engine/crates/layerfs-remote-storage (no first-party deps; only sha2, already locked): driver.rs (limits, upload_file of a file or span, download_file whole/ranged into a writer, stat, delete), limits.rs, pieces.rs, error.rs, credentials.rs; features local_folder, s3_api, minio.
   • s3_api/: the retired one-attempt SigV4 client revived, plus streamed bodies, canonical query strings, multipart (create, equal parts 5 MiB–5 GiB, ≤10,000 parts, complete; one abort on any failure) and ranged GET.
   • minio/: 50 TiB object, 5 TiB single PUT, 64 MiB default parts; LAYERFS_REMOTE_* settings or a credential reference.
   • layerfs-store-service/src/remote/: settings.rs (named remotes, credential reference only), push.rs (OpenedStore::snapshot from session a → limits check → upload with sha256), pull.rs (download to store.sqlite.pulling → sha256 → opens with the project's binding → rename over the local Store).

   Checks

   Check	Where	Result
   layerfs-remote-storage --test folder_pieces (5)	macOS	pass
   layerfs-remote-storage --test minio --include-ignored (7) against pinned MinIO	macOS + Docker	pass
   layerfs-store-service --test remote_settings (2)	macOS	pass
   layerfs-api --test remote_store: folder_round_trip + minio_round_trip	Linux (Docker)	pass
   Clippy -D warnings on the touched packages	Linux + macOS	clean

   MinIO image: cgr.dev/chainguard/minio@sha256:4692462f35d97d7e82c30371d82f057703c5d9489bcae726010594c812f2d285 (RELEASE.2026-09-22T19-25-18Z). The community repo is archived; the image is used only for tests (tests/minio_service.sh).

   The MinIO tests cover:

   • multipart above the part size (create, 3 parts, complete);
   • a refused part, aborted with no parts left;
   • a lost part reply: Uncertain, part sent once, then aborted;
   • a failed abort: reported PartsLeft and listed by pending_uploads.

   The Store round trip pushes a 12 MiB+ Store (multipart) with three Branches and four Commits, then pulls it into a fresh volume. Branches, Commit lists and file reads are identical. Also checked:

   • a Commit made after the push is not in the pulled Store;
   • an open Store is not replaced, and a closed one is;
   • another project's binding is refused.

   Decisions and deviations

   1. "No opener holds it" is decided by SQLite's -wal/-shm sidecars. On Linux they exist exactly while the Store is open, and pull only runs on Linux behind the kernel guard. There's no SQLite code in store-service. The check doesn't stop a new opener from arriving before the rename: the caller stops the project's daemons and SDK processes first.
   2. Pull follows the install rules in install.rs: kernel guard on the volume, run as the daemons' user (uid 0). installer_user became pub(crate). It doesn't call install_local itself, because that checks for an initial Branch with no Commit, and a pulled Store has history.
   3. One TCP connection per request (Connection: close) rather than keep-alive. A failure then can't leak into the next request, and an abort after an uncertain part gets a fresh connection. Socket I/O has per-operation timeouts instead of one deadline per request, so multi-GB parts work.
   4. Each body's sha256 is signed (computed before the request), so MinIO rejects changed bytes. The cost is reading each part twice.
   5. Pieces are uploaded before the manifest, which goes under the Store's own name. Stale pieces from an earlier, larger push are removed afterwards. With no compare-and-swap, a pull racing a push fails its sha256 check.
   6. Settings are a caller-placed text file (Remotes::load/save). There are no SDK methods or layerfs command for remotes yet.
   7. The MinIO service tests are #[ignore]d and run explicitly against tests/minio_service.sh. tools/worktree.py cargo runs Linux tests with --network none, so I ran the Linux MinIO round trip by starting the launcher-built test binary in a container on MinIO's Docker network. Same target volume and build image.
   8. Tooling: the crate was added to the boundary guard's layer map and to production_loc.py's engine list.

   Not in this phase

   • TLS (needed for R2 and AWS) and the other providers.
   • SDK methods and a CLI for remotes.
   • Incremental push (feat(persistence): export a Store's changed rows as bundles for replicas #495).

   Open item for area A

   After a9903e1286, the boundary guard reports backend/sqlite/lifecycle/file_control.rs as unsafe outside the audited path. Session a was told.

   Production LOC (R commits): 102997 → 104696 (+1699, new crate) → 105080 (+384, store-service remote/) → 105080 (0).
3. yifanxuaaa commented on Oct 29, 2026

   

   yifanxuaaa

   on Oct 29, 2026

   ContributorAuthor

   More actions

   Owner direction (2026-10-29): one Store holds many projects, and a home holds many Stores (local SQLite Stores, PostgreSQL connections, remote sources). Design doc committed as 69e2909: core/docs/issues/506/README.md. Push, pull and clone now work per Store, and each PostgreSQL Store is one schema. Part c (Store-first API layer) goes to a third session. Progress so far: 514a79f snapshot, a9903e1 SQLite code grouped by role, 5586d8f remote drivers (folder, MinIO), b55e317 push/pull, 9c6ebef remote docs.
4. yifanxuaaa

   added a commit that references this issue on Oct 29, 2026

   feat(remote): Store-named remotes per home, transfer state, open and …

   ...

   6ad83a6
5. yifanxuaaa commented on Oct 29, 2026

   

   yifanxuaaa

   on Oct 29, 2026

   ContributorAuthor

   More actions

   Area R: Store-first adjustment (6ad83a6027) — session b

   Following the parent session's design adjustment and the owner's correction: push and pull now work on a Store, with every project in it. A home can hold many Stores.

   Request	Built	Difference
   Remotes once per home; key <prefix>/<store>/store.sqlite	HOME/remotes (Remotes::home_path); store_object(name) gives NAME/store.sqlite, plus pieces	Store names are one segment of [A-Za-z0-9._-], at most 128 characters, not starting with a dot
   Push/pull state outside the Store, keyed by Store	remote::state: HOME/remote-state/NAME holds remote, direction, time, sha256 and size	push and pull don't take the home, so the caller saves a Transfer after each completes. This keeps them usable without a home
   Pull refuses while a daemon, mounted Workspace or SDK/worker holds the Store; temporary file on the same filesystem	Refused before the download and again before the rename. The error names those openers and the macOS worker's quiet minute. The download is store.sqlite.pulling beside the Store, so the swap is one rename	Equivalent, not by listing holders: every opener holds SQLite's -wal/-shm, including the macOS worker, which runs in a Linux container. Nothing stops a new opener between the check and the rename
   Push stages beside the Store; free-space check with a clear error	push(opened, store, driver, name) stages store.sqlite.pushing beside the Store. PushFailure::Space names the directory, bytes needed and bytes free	The amount needed is the main file + WAL plus the Store's 64 MiB write headroom (STORE_HEADROOM_BYTES), so a push can't starve the live Store. No automated test (it needs a full filesystem)
   Any endpoint URL, no localhost assumption	Any host name (container names with _), IPv4 or bracketed IPv6, optional trailing /. minio_service.sh also prints the in-network endpoint	The Linux MinIO round trip now reaches MinIO by container name (http://lfs506-minio:9000)

   Checks:

   • macOS: remote-storage minio (7, against MinIO), folder_pieces (5), store-service remote_settings (3).
   • Linux: remote_store folder and MinIO round trips, now with two Stores on one remote, push/pull state recorded, and an open Store refused before any download.
   • Clippy -D warnings on Linux and macOS.

   Production LOC: 105080 → 105282 (+202).
6. yifanxuaaa

   added a commit that references this issue on Oct 29, 2026

   feat(remote): clone a Store into a new one; push records the binding …

   ...

   7247d27
7. yifanxuaaa commented on Oct 29, 2026

   

   yifanxuaaa

   on Oct 29, 2026

   ContributorAuthor

   More actions

   Area R: clone (7247d277cd) — session b

   This adds clone for the Store-first API layer (session c), following the design doc (core/docs/issues/506/README.md §3, §6).

   • remote::clone_store(driver, store_name, destination, cursor_key) pulls a Store into a new Store.
     • The destination must not exist yet. The file is hard-linked into place, so a Store that appears there in the meantime is never replaced.
     • The binding key is read from the remote copy and returned.
     • cursor_key is the new Store's own authority. This works because opening a Store checks only the binding.
   • Push records the binding as layerfs-binding metadata (hex). The binding isn't secret; the cursor key never leaves its holder. A pull now refuses a remote copy with another Store's binding before downloading anything.
   • The push precondition comes from §3's two-machine flow. Expected::{Any, Absent, Sha256}, taken from the home's last transfer, must match the remote copy; otherwise push refuses with RemoteMoved. It's checked before the snapshot and again just before the upload.
     • Not atomic. The owner requires no compare-and-swap, so a push by another machine between the check and the upload isn't detected.

   Checks: on Linux, the folder and MinIO round trips now include a clone with identical history, a refused clone over an existing Store, refused pushes after the remote moved (nothing uploaded), and a refused pull of another binding. Clippy -D warnings passes on Linux and macOS.

   Production LOC: 105282 → 105436 (+154).
8. yifanxuaaa

   added 3 commits that reference this issue on Oct 29, 2026

   feat(persistence): PostgreSQL Store engine behind one seam (#506 A2)

   ...

   c94749e

   feat(persistence): import a SQLite Store into PostgreSQL; identical i…

   ...

   ee7cd91

   fix(persistence): keep entry files declarative; guard the moved FFI p…

   ...

   f84f1d2
9. yifanxuaaa commented on Oct 29, 2026

   

   yifanxuaaa

   on Oct 29, 2026

   Last edited by yifanxuaaa

   ContributorAuthor

   More actions

   Area A (persistence): fix record, session a

   Branch issue506-store-backends-remote, PR #507. Six commits, all on top of b's work:

   Commit	What	Production LOC
   514a79fdcd	Snapshot of a live SQLite Store (online backup API, one step)	102831 -> 102950 (+119)
   a9903e1286	A1: SQLite code grouped by role under backend/sqlite/, no behaviour change	102950 -> 102997 (+47)
   c94749e591	A2: PostgreSQL engine behind one seam (feature postgresql)	105436 -> 107114 (+1678)
   ee7cd91365	A4: import a SQLite Store into PostgreSQL; identical-ids test	107114 -> 107544 (+430)
   f84f1d25f5	Boundary-guard fix: entry files stay declarative; guard names the moved FFI path	107544 -> 107547 (+3)
   3092604cb0	rustfmt the PostgreSQL open test (test only)	107547 -> 107547 (0)

   The gap between a9903e1286 and c94749e591 is b's remote commits. LOC is from tools/production_loc.py, first parent vs. commit tree. No tests were deleted or moved.

   Layout at f84f1d25f5

   engine/crates/layerfs-persistence/src/backend/ only, using tools/production_loc.py --files:

     backend/                                    [6815]
       admission.rs                                  118   (moved from sqlite/lane/)
       mod.rs                                         11
       records.rs                                    158
   +   seam.rs                                        35   (new: Engine, Mechanics, HistoryStatements)
   +   postgresql/                                [1560]   (new, #506 A2/A4)
   +     import.rs                                   399
   +     mechanics.rs                                 51
   +     mod.rs                                        9
   +     schema.rs                                    88
   +     history/                                   [13]
   +       mod.rs                                      1
   +       statements.rs                              12
   +     lane/                                      [56]
   +       lane.rs                                    54
   +       mod.rs                                      2
   +     metadata/                                 [169]
   +       mod.rs                                      2
   +       reads.rs                                  123
   +       writes.rs                                  44
   +     objects/                                  [208]
   +       mod.rs                                      2
   +       publish.rs                                 95
   +       read.rs                                   111
   +     session/                                  [468]
   +       connection.rs                              91
   +       mod.rs                                      5
   +       profile.rs                                 59
   +       query.rs                                   31
   +       rows.rs                                    62
   +       transaction.rs                            220
   +     storage/                                   [99]
   +       mod.rs                                      1
   +       provider.rs                                98
       sqlite/                                    [4933]   (A1: grouped by role)
   +     mechanics.rs                                 51   (new: SQLite behind the seam)
         mod.rs                                       10
         schema.rs                                   295
   +     snapshot.rs                                 100   (new, #506)
         acquisition/                             [1527]
           accounting.rs                              74
           cleanup.rs                                190
           entry_windows.rs                          236
           metadata.rs                                41
           mod.rs                                     11
           provider.rs                               246   (from src/storage/acquisition/)
           reads.rs                                  364
           root_windows.rs                            56
           statements.rs                              86
           writes.rs                                 223
         lane/                                     [124]
           lane.rs                                    42
           mod.rs                                      3
           writer_admission.rs                        79
         lifecycle/                                [311]
           file_control.rs                            30
           headroom.rs                                13
           mod.rs                                      9
           reclamation.rs                             48
           seal.rs                                    78
           seal_allocation.rs                        133
         metadata/                                 [316]
           allocation.rs                              19
           locations.rs                              129
           mod.rs                                      5
           policy.rs                                  18
           signatures.rs                              77
           values.rs                                  68
         objects/                                  [561]
           mod.rs                                      7
           publish.rs                                135
           read.rs                                    65
           selection.rs                               83
           unit_io.rs                                 70
           unit_layout.rs                             37
           units_publish.rs                           34
           units_read.rs                             130
         session/                                 [1165]
           connection.rs                             207
           mod.rs                                      8
           prepared.rs                               166
           profile.rs                                104
           query.rs                                  206
           rows.rs                                    57
           statement_work.rs                          33
           thread_work.rs                             84
           transaction.rs                            300
         storage/                                  [473]   (merge_records from src/storage/)
           merge_records.rs                          472
           mod.rs                                      1

   What is in place

   • Snapshot. Handles::snapshot, StorageProvider::snapshot and store-service OpenedStore::snapshot.
     • It copies a live Store on its own read-only connection in one backup.step(-1).
     • The new file is created exclusively. It is WAL, truncate-checkpointed and closed with no sidecar.
     • Session b's push uploads it.
   • Seam. History, Metadata/Objects bounds and publication validation stay shared. They reach an engine only through Mechanics and HistoryStatements (backend/seam.rs).
     • The shipped SQLite history SQL runs on PostgreSQL unchanged, except hint_insert_binding and statistics_totals.
   • PostgreSQL engine (postgres =0.19.14, TLS through postgres-native-tls/native-tls; all optional behind postgresql).
     • BackendSelection::Postgres(PostgresSelection { url, schema, synchronous_commit, lock_timeout_ms }).
       • synchronous_commit can be off (default), local, remote_write, on or remote_apply.
       • It is applied with SET LOCAL in the same round trip as BEGIN.
       • Debug output redacts the URL.
     • Transactions: reads are REPEATABLE READ READ ONLY; writes are READ COMMITTED. Never SERIALIZABLE, never retried.
     • Lane: a lock timeout is Busy. A lost COMMIT reply is Unknown and quarantines the session.
     • Packs: bodies go in with COPY BINARY before the lane. They are bytea STORAGE EXTERNAL, and ranged reads use substring.
     • Opening validates the schema version and binding. It never creates or migrates anything.
     • Store-service has StoreService::open_postgres. The bridge has ProviderKind::Postgres (wire 2), which an install manifest refuses.
   • Import. PostgresHandles::import(sqlite_config, selection, binding, cursor).
     • It creates the schema and copies every table and pack body (any of the three SQLite pack layouts) in one PostgreSQL transaction, over one SQLite snapshot.
     • It keeps object ids, roots, Commit ids and inode serials.
     • Init stays SQLite only.
   • Identical ids (layerfs-persistence/tests/postgresql_identical_ids.rs).
     • Setup: a Store with two Commits on a Branch is imported into PostgreSQL 17.11.
     • Image: postgres@sha256:639ab7ceb90e13123085b741fb31ef493fba25463002f6da665352e7b534b652.
     • The test publishes the same third Commit on both engines and compares object, root and Commit ids, plus reads.
     • A second test checks that a held lane makes another process's write Busy after lock_timeout.
     • The store-service test postgresql_open.rs opens the imported Store through StoreService::open_postgres.
     • The tests run only when LAYERFS_TEST_POSTGRES_URL is set; otherwise they report as skipped.

   Deviations (my calls, under the owner's full authority)

   1. The lane is per Store, not per head row. Every write locks one store_lane row FOR UPDATE. This matches SQLite (one writer per Store), so projects in one PostgreSQL Store still queue behind each other. README §2.2 asks for per-head locking.
      • Per-head locking is the next step. It needs FOR UPDATE on head reads, plus a review of name checks, the scope-allocator CAS and lineage writes. These currently rely on the single writer.
      • A held publication lane owns the row on a second connection. That thread's nested writes skip the wait.
   2. Packs are stored whole, not sliced. Each pack body is one bytea value (EXTERNAL, so TOAST slices it and substring reads only the range). Splitting into slice rows can follow if a pack goes over PostgreSQL's 1 GB value limit.
   3. Not on PostgreSQL in this phase:
      • Merge records: merges are refused, so merge gives BackendUnavailable.
      • Init and acquisition.
      • Seal, snapshot and reclamation, since there is no file.
      • The reader counters in store diagnostics are not observed.
   4. The PostgreSQL schema has its own version (1). Import rewrites schema_version, and the SQLite versions 7–12 do not apply there.
   5. No kernel guard on PostgreSQL. StoreService::open_postgres skips the Linux kernel/filesystem guard that SQLite opening keeps, because the Store is not a local file.
   6. The import opens the SQLite source writable. Handles::open_read_only cannot open a sealed or snapshot Store (see finding 1). The import does not write to it; it reads from one consistent read transaction.
   7. What stays shared. src/history/ and src/storage/publication.rs hold no SQL, so they stay shared instead of moving under backend/sqlite/. Writer admission moved up to backend/admission.rs, where both engines use it.

   Findings

   1. Handles::open_read_only cannot open a closed WAL Store whose -shm is gone (a sealed Init output or a snapshot): it fails with Uncertain. Opening read-write once fixes it. This matters for any read-only consumer of pulled or cloned files; session b's clone opens writable.
   2. PostgreSQL needs explicit casts on parameter arithmetic ($1::bigint + $2::bigint). Otherwise the shared history SQL fails with operator is not unique. The changed statements carry the casts.

   Checks (macOS, at f84f1d25f5)

   • cargo +1.85.1 fmt --all --check. At f84f1d25f5 it missed tests/postgresql_open.rs, which CI reported; 3092604cb0 formats it.
   • clippy -D warnings on persistence, bridge and store-service, with and without postgresql.
     • Store-service is clean at 740fd07615. An earlier useless_conversion report came from an in-between edit of b's that never landed.
   • Persistence suite with and without the feature: with the feature and the container, 121 tests passed and none failed.
   • Store-service and bridge tests.
   • cargo check --workspace.
   • tools/check_product_boundary.py PASS, plus its 34 self-tests.
   • Ruff format.

   Every test command ran under a 120 s wall limit. SQLite tests used the Disposable profile only.

   Gaps and next

   • The persistence and store-service suites have not run on Linux.
     • Session c's Linux run (#506 comment) built the postgresql feature on Linux, native-tls included.
     • It covered import, attach through PostgresHandles::binding, and remove_layerstack against PostgreSQL 17, in the Linux runtime image on Docker Desktop, not a bare Linux host.
   • The host SQLite cache for PostgreSQL Stores (backend/postgresql/cache/, PostgreSQL compatibility for the global storage #420) is next. Without it a Commit takes several round trips.
   • Per-head lane locking (deviation 1): accepted for phase one by the coordinating session; it stays an open item, not this phase's work.
   • Merges on PostgreSQL, and sliced packs if needed.
   • The new PostgreSQL dependencies bring a wasm-bindgen family bump in Cargo.lock, through getrandom/js-sys. It is harmless on native targets; noted for review.
10. yifanxuaaa

    added 3 commits that reference this issue on Oct 29, 2026

    feat(project): add a project to a live Store; list a Store's projects (…

    ...

    b0fd281

    style(store-service): rustfmt the PostgreSQL open test (#506)

    ...

    3092604

    feat(persistence): remove one project from a Store; read a PostgreSQL…

    ...

    740fd07
11. yifanxuaaa commented on Oct 29, 2026

    

    yifanxuaaa

    on Oct 29, 2026

    ContributorAuthor

    More actions

    Area A: the persistence API that session c builds on (at 740fd07615)

    Everything PostgreSQL is behind the Cargo feature postgresql (persistence, and store-service's feature that forwards to it).

    Select and open a PostgreSQL Store. A Store is one schema holding all of its projects.

    • Selection: PostgresSelection::new(url, schema).
      • .with_synchronous_commit(SynchronousCommit::{Off (default), Local, RemoteWrite, On, RemoteApply}).
      • Public field lock_timeout_ms (default 30000).
      • It goes into BackendSelection::Postgres(..).
    • PostgresHandles::open_writable(&selection, binding, cursor_key) and open_read_only(..).
      • The result is PostgresHandles { storage: Arc<PostgresStorage>, history: HistoryProvider }.
      • Opening validates; it never creates or migrates.
    • PostgresHandles::binding(&selection) -> Result<Vec<u8>, PersistenceError>: the binding key a LayerFS schema records, for attaching a schema this home did not import.
    • Store-service: StoreService::open_postgres(..), the same open path as SQLite. OpenedStore::snapshot there is SQLite only.

    Import. PostgresHandles::import(sqlite_config, &selection, binding, cursor_key) -> Result<ImportReceipt { tables, pack_bytes }, _>.

    • The target schema must not exist.
    • It copies the whole SQLite Store, every layer stack, in one PostgreSQL transaction, keeping every id.

    Remove a project. HistoryCatalog::remove_layerstack(stack) -> HistoryResult<()>, on both engines.

    • One write transaction deletes the stack's Layers, Branches and Commits, their lineage, and their logical bytes from the totals.
    • Packs, objects and the allocator scope stay; reclaiming packs is layerfs rm/compact: retire Branches and reclaim unreachable Store history #411.
    • An unknown stack is Missing(LayerStack). Other providers refuse it as Unsupported.
    • The write holds the writer for time that grows with the removed project's history. Batching can follow if that matters.

    Snapshot. Handles::snapshot(dest) and OpenedStore::snapshot(dest) -> SealedStore, SQLite only.

    None of these signatures is planned to change. As recorded in the fix record above, the PostgreSQL lane still locks per Store, not per head row.
12. 8 remaining items

    Load more
13. yifanxuaaa

    added a commit that references this issue on Oct 29, 2026

    feat(remote): Cloudflare R2 over HTTPS; TLS for the S3 API driver (#506…

    ...

    809c86f
14. yifanxuaaa commented on Oct 29, 2026

    

    yifanxuaaa

    on Oct 29, 2026

    ContributorAuthor

    More actions

    Cloudflare R2 over HTTPS: 809c86fe9b

    The S3 API driver now does HTTPS, and cloudflare_r2/ is the second remote provider.

    • TLS is a tls feature on remote-storage, store-service and layerfs-node. It is off by default, like postgresql. It uses the native-tls already pinned for PostgreSQL, so it adds no new package. The server's certificate is checked against the system's roots, and the TLS handshake happens before the request is sent. A refused certificate is therefore Unavailable, with nothing sent.
    • An R2 remote is { provider: 'r2', account, jurisdiction?, bucket, prefix? }. It is reached at https://ACCOUNT[.eu|.fedramp].r2.cloudflarestorage.com, path-style, with region auto. Its limits are 4.995 TiB per object and 4.995 GiB per single PUT; larger files use multipart.
    • Worker build: --features layerfs-node/postgresql,layerfs-node/tls.
    • Connect (not committed) has an R2 form: account ID, jurisdiction, bucket, prefix and keys. The keys stay in Connect. Connect also has an R2 page that works like the MinIO one.

    Checked:

    • Clippy is clean on macOS, with and without tls.
    • remote_settings passes 4/4.
    • On Linux, minio.rs passes 8/8 over plain HTTP. It also passes 8/8 against a TLS MinIO started with the new MINIO_TLS_DIR option. The four fault-proxy tests read plain HTTP, so in the TLS run they say they were skipped. A new direct multipart test covers multipart over TLS.
    • An untrusted certificate is refused at the handshake.
    • Through the macOS worker, an R2 remote is added with its derived endpoint. A push with fake keys to a made-up account reaches Cloudflare. Cloudflare ends the TLS handshake for an unknown account (alert 40), and curl gets the same.

    NOT_RUN: a real R2 account. This needs an account ID plus an R2 API token's S3 access key and secret, with Object Read & Write.
15. yifanxuaaa commented on Oct 29, 2026

    

    yifanxuaaa

    on Oct 29, 2026

    ContributorAuthor

    More actions

    #506 d: Sandboxes on PostgreSQL Stores (plan)

    Session d, branch issue506-pg-sandbox from issue506-store-backends-remote (809c86fe9b); merges back there (draft PR #507).

    Goal. A Sandbox's daemon serves a project whose Store is a PostgreSQL schema, so Workspaces mount, Exec runs and Commits publish into that schema, the same as on SQLite. Then Connect can run scenarios and live sessions on PostgreSQL projects.

    Already in place. layerfs-store-service's open_store_observed opens a PostgreSQL Store when its PersistenceConfig.backend is Postgres(selection). The SDK's in-process Store calls already use it. The daemon only ever passes PersistenceConfig::sqlite(&setup.store), and the Docker layer always mounts a Store volume.

    1. How the daemon learns its Store is PostgreSQL

    DaemonSetup gains postgresql: Option<DaemonPostgres>, with { url, schema, synchronous_commit, binding, cursor_key }. It cannot be set together with existing_store. The config's magic goes from LFSD\x05 to \x06. The daemon and the binding come from one bundle, so no old reader exists.

    At startup the daemon builds PostgresSelection::new(url, schema).with_synchronous_commit(..) and opens it through the same open_store_observed. The default 30 s lane lock_timeout stays. The daemon then starts ControlReady, as an existing-Store daemon does, with no install.

    • The binding and cursor key come from the home's store.record, the same ones the SDK's Store calls use.
    • synchronous_commit is the Store record's value: off by default, or whatever attach_postgresql set.
    • layerfs-daemon gets a postgresql feature that forwards to layerfs-store-service/postgresql. The bundle builds with --features layerfs-node/postgresql,layerfs-daemon/postgresql. A daemon built without it refuses at startup with BackendUnavailable.

    2. Keeping the URL away from agent processes

    The URL, password included, travels only inside the daemon's private config file. That file is uploaded through the Engine's archive API to /layerfs-local/config/daemon.setup. The file is mode 0600, owned by the daemon's uid, in a 0700 directory. The daemon checks those modes before it reads the file.

    The URL is never placed in container env, argv, labels, the Sandbox record or a log:

    • Debug redacts it.
    • The daemon makes itself non-dumpable.
    • Commands run as uid 501/gid 20 with no capabilities.

    So an agent process in the Sandbox cannot read the file, /proc/<daemon>/{environ,mem,fd}, or raw packets. The host never writes the URL to disk. sandbox.create takes it per call from the caller's postgresql: { url }, the same field every Store call already carries from LAYERFS_STORE_POSTGRESQL.

    3. Network reach

    The daemon dials the host and port in the URL from the Sandbox container's default bridge network. On Docker Desktop (macOS, Windows, and the E2E's linux mode), host.docker.internal reaches a server published on the host's loopback. That is the same URL the worker and the client already use.

    On a Linux Docker Engine the URL must name an address that containers can reach, not 127.0.0.1. This phase adds no ExtraHosts mapping and no second "daemon URL". It is documented and recorded as an open item.

    4. Overlay

    Each Workspace overlay stays its own local SQLite file: MEMORY/OFF/EXCLUSIVE, copied from the template at mount and deleted at unmount, per the owner model. Only the global Store moves to PostgreSQL.

    A PostgreSQL Sandbox mounts no Store volume. SandboxRequest.store_volume becomes Option<String>, and None means the container has no mounts at all. The topology check on reopen then expects zero mounts, and the Sandbox record stores no volume.

    5. Demand-loaded reads and Commit round trips

    This phase takes the cost and adds no host cache. The daemon's existing 64 MiB shared immutable-object cache covers PostgreSQL reads too, so repeat reads across Workspaces in one Sandbox stay local. A cold read or a Commit pays the round trips #420 measured (about 873 per Commit without a host cache). On a loopback or Docker Desktop link that is tens to hundreds of milliseconds per Commit.

    The publish step itself is the existing per-Store FOR UPDATE lane: heavy work happens before it, and only the publish holds it. The host cache stays §8's open item (#420). The E2E prints the Commit's wall time as a diagnostic only, not as a gate or a measurement.

    Inherited limit: merges are refused on PostgreSQL in this phase. A Commit whose Branch moved since mount, and so needs a merge, fails instead of merging. This is the same limit the SDK path has today.

    6. Runtime and binding calls

    • sandbox.create (binding). For a project of a PostgreSQL Store, the call reads the Store record and the call's postgresql.url, and checks that the project is in the schema. It then starts the container with no volume and setup.postgresql set. The container name is <prefix><store>.<project>-<n>, labelled layerfs.store. deploy.volume is not needed. Without postgresql.url the call is refused InvalidInput.
    • Runtime known(). No longer refuses PostgreSQL projects. get_state(name, since, { store, url }) and sandbox.reclaim(name, { store, url }) take the URL per call. Without it they are refused InvalidInput.
    • Runtime clients. A PostgreSQL project's client is opened for that one call, with LAYERFS_STORE_POSTGRESQL, and closed after it. It is not cached for a minute, so the runtime never keeps the URL.
    • Agents. @layerfs/agent with LAYERFS_STORE and LAYERFS_STORE_POSTGRESQL can sandbox.create, workspace.mount/exec/commit/unmount, and sandbox.close, with deployment settings as for SQLite.
    • Doc. §6 drops "PostgreSQL Stores have no Sandboxes" and states the above. §8 gains the Linux-Engine reach item.

    Done when

    api/runtime/test/verify/stores_real.mjs gains a step on the imported PostgreSQL project. It runs sandbox.create, mounts feature, runs one command, Commits, and unmounts. It then reads the Commit back through the PostgreSQL Store (branch.list_commits and fs.read). The step runs on linux and, if time allows, on macos.

    Every commit carries its production LOC line.

    Owner decisions needed

    None found. The plan stays inside the owner model:

    • one global DB with one writer and short holds;
    • each overlay stays SQLite;
    • no new dependency (postgres and native-tls are already pinned);
    • no retry;
    • the URL is never stored.
16. yifanxuaaa

    added a commit that references this issue on Oct 29, 2026

    feat(daemon): Sandboxes on PostgreSQL Stores (#506 d)

    ...

    757b91b
17. yifanxuaaa commented on Oct 29, 2026

    

    yifanxuaaa

    on Oct 29, 2026

    ContributorAuthor

    More actions

    #506 d: Sandboxes on PostgreSQL Stores — done

    Commit 757b91b48e on issue506-store-backends-remote (fast-forward from 809c86fe9b), draft PR #507. Built as the plan says. No owner decision was needed.

    What works now. A Sandbox's daemon serves a project of a PostgreSQL Store. sandbox.create, workspace.mount, exec, commit, unmount and sandbox.close all behave as on SQLite. A Commit publishes into the schema, and the SDK reads it back.

    How it is built

    • Daemon config. The bridge's DaemonSetup gains postgresql: Option<DaemonPostgres>, holding the URL, schema, synchronous_commit, binding and cursor key. It cannot be set together with existing_store, and the config magic is now LFSD\x06. The URL lives only in the daemon's private config file (0600, daemon uid, 0700 directory, non-dumpable daemon). Debug redacts it.
    • Daemon startup. The daemon opens the schema through open_store_observed and starts ControlReady, with no install and no Store-file checks.
      • Its status reports the engine as postgresql/<version>. The status wire allows one printable word, and PostgreSQL's server_version contains spaces; this was the first E2E failure.
      • New feature layerfs-daemon/postgresql.
    • Docker. SandboxRequest.store_volume is now Option. None mounts nothing, and the reopen topology check then expects zero mounts.
    • Binding. sandbox.create on a PostgreSQL Store reads the home's Store record and the call's postgresql.url. It checks that the project is in the schema and makes its home entry (owner-only).
    • Agent. No LAYERFS_VOLUME is needed.
    • Runtime.
      • get_state(name, since, { store, url }) and sandbox.reclaim(name, { store, url }) take the URL per call. Without it they are refused InvalidInput.
      • A PostgreSQL project's client is opened for the call and closed after it.
      • The open-time sweep still skips PostgreSQL projects, because it holds no URL.

    Checks

    Check	Result
    stores_real.mjs linux	MAIN ok. New step: mount feature, exec, Commit, read back (Commit count +1, file content). The step also asserts that /proc/mounts has no /layerfs-store and that no command output holds the password. Commit wall time 158 ms (diagnostic only).
    stores_real.mjs macos (worker)	MAIN ok. Commit 160 ms (diagnostic only).
    Linux Clippy -D warnings, both features, --all-targets	clean: bridge, sandbox, daemon, api, node
    Host tests	bridge (incl. new daemon_setup_records), sandbox (incl. no-volume engine_protocol case), node, api sandbox_operations: all pass. native_admission was not run (owner-excluded).
    Runtime stand-in cases (15), agent calls	PASS. The stores case gains the PostgreSQL get_state URL check.
    check_product_boundary.py	PASS

    The bundle was build-linux-d20dfdc3a53f4071ab7bbcfbed1405da, built with --features layerfs-node/postgresql,layerfs-daemon/postgresql, runtime image sha256:afa1ef9c…, PostgreSQL postgres:17-bookworm (sha256:639ab7ce…). Load average was 4.5–4.9.

    Not run: the daemon's FUSE mount-lane tests (layerfs-daemon, the API mount targets). The E2E covers the daemon on both Store kinds.

    Production LOC: 109,626 → 109,881 (delta +255). By scope:

    • core: 38,400 → 38,400 (unchanged)
    • engine: 68,159 → 68,388 (+229)
    • api: 3,067 → 3,093 (+26)

    The reference is absent (retired R9c; last counted 65,417 at tree 498dd1917812). Counted with tools/production_loc.py on git archive of the parent and of the commit.

    For Connect

    • It can drop the "Sandboxes do not open PostgreSQL Stores yet" refusal.
    • It passes { url } to runtime.project.get_state and runtime.sandbox.reclaim for PostgreSQL projects.
    • Its agent clients for those projects keep LAYERFS_STORE_POSTGRESQL, as open() already does.
    • The worker and daemon bundle must be built with both features.

    Open (in §8):

    • No host SQLite cache yet (PostgreSQL compatibility for the global storage #420): cold reads and Commits pay server round trips.
    • Merges are refused on PostgreSQL, so a Commit whose Branch moved fails instead of merging.
    • On a Linux Docker Engine, the URL must name an address that containers can reach; no host-gateway mapping is added.
18. yifanxuaaa

    added 3 commits that reference this issue on Oct 29, 2026

    feat(persistence): merges on PostgreSQL Stores (#506 d)

    ...

    50335fd

    perf(persistence): fewer PostgreSQL round trips per Commit and read (#…

    ...

    207553a

    docs(506): PostgreSQL round trips and reaching a server container (#506…

    ...

    a73d07c
19. yifanxuaaa commented on Oct 29, 2026

    

    yifanxuaaa

    on Oct 29, 2026

    ContributorAuthor

    More actions

    #506 d: Connect scenarios on a PostgreSQL Store — all 11 pass

    Commits on issue506-store-backends-remote (fast-forward from 757b91b48e, draft PR #507):

    • 50335fd5d4 feat(persistence): merges on PostgreSQL Stores. A merge's working records live in pg_temp tables of the writer connection. Production LOC 109,881 → 110,184 (+303).
    • 207553a84a perf(persistence): fewer PostgreSQL round trips. Production LOC 110,184 → 110,381 (+197).
    • a73d07c90a docs(506): doc §6 note. Production LOC 110,381 → 110,381 (delta 0).

    The reference is absent (retired R9c; last counted 65,417 at tree 498dd1917812).

    Why depot and crew ran past 15 minutes

    I profiled a copy of the Store with pg_stat_statements. The time was round trips, about 0.75 ms each through Docker Desktop's host.docker.internal. The causes:

    • Per-row statements: 11,398 binding_hint inserts and 9,532 file_digest lookups.
    • BEGIN + SET LOCAL + COMMIT around every immutable read.
    • A separate descriptor query before every ranged pack read.

    One Commit issued 39,555 statements, and one fs.apply took 95.8 s.

    Fixes

    • Array statements. Binding hints and file digests are written as array (unnest) statements on PostgreSQL: up to 1,024 rows per statement, and one statement per 64-content digest window. When a digest repeats, the later root wins, as it would one row at a time. SQLite keeps its per-row statements.
    • Immutable reads outside a transaction. Locations, inode values and packs are read outside a transaction, as on SQLite. Their statements name the schema.
    • Pack reads. A pack read takes its descriptor and the first 64 KiB of the body in one statement. Whole-pack reads take every checked body in one statement.
    • Connect, uncommitted in the private repo. A loopback PostgreSQL URL whose port a container on the default bridge publishes is handed to the worker and Sandboxes at that container's bridge address. A round trip there costs about 0.13 ms instead of 0.75 ms. Anything else keeps host.docker.internal.

    Result: the worst fs.apply went from 95.8 s to 0.4 s. One Commit went from 39,555 statements to about 3,300.

    Runs (Connect test host, macOS worker, one sample each)

    Scenario	Result	Time
    tour	passed 14/14	38 s
    parallel	passed 14/14	29 s
    conflicts	passed 14/14	37 s
    merge	passed 14/14	36 s
    sandbox	passed 7/7	34 s
    agents	passed 199/199	151 s
    teams	passed 17/17	33 s
    shop	passed 18/18	69 s
    depot	passed 98/98	560 s
    crew	passed 967/967	462 s
    checkpoints	passed 37/37	56 s

    Setup for these runs:

    • Release bundle build-linux-f0296dc4de7f4cb3bd47a51dbcad5b73 at 207553a84a, built with layerfs-node/postgresql,layerfs-daemon/postgresql.
    • PostgreSQL 17 in a container, synchronous_commit=on.
    • 1-minute load average 5.9–8.3.

    SQLite vs PostgreSQL with the debug bundle. I ran depot on PostgreSQL and on an empty SQLite project side by side, with the same debug bundle and host. Both reached exactly 53/98 steps at 702 s, and I stopped both. So with the debug bundle on this loaded host, depot misses 15 minutes on either engine; it is not a PostgreSQL limit.

    Checks

    • New test hints_and_digests_in_arrays_read_back_as_on_sqlite: 1,510 hints, 1,100 digests across the 1,024-row boundary, and a missing root. It passes, as do the existing PostgreSQL tests.
    • layerfs-persistence crate tests pass, all 40 binaries.
    • Linux Clippy with -D warnings is clean for persistence and store-service with postgresql.
    • Connect's storage.test.js passes 13/13, including a new route test.

    Open

    • There is still no host cache (PostgreSQL compatibility for the global storage #420). Cold reads in each new Sandbox pay round trips. With a remote server (higher RTT), heavy scenarios will be slower.
    • The Connect route covers only a server container on the same Docker engine's default bridge.
20. yifanxuaaa

    added 4 commits that reference this issue on Oct 29, 2026

    feat(remote,persistence): progress, listing and heads for Connect Sto…

    ...

    2740731

    feat(store-service,binding): heads, copy checks, listing, progress, P…

    ...

    8b00141

    feat(runtime): Store sizes, heads, copy checks, listing, progress and…

    ...

    d67e1e8

    style(remote): rustfmt the MinIO listing test (#506)

    ...

    00665aa
21. yifanxuaaa

    added sub-issues on Oct 29, 2026

    • #506 Stores: one link per Store, connections bind to providers only #509
    • #506 Live view: browse a Store in S3 without loading it #510