Repository navigation
#506: Store backends (SQLite + PostgreSQL) and remote Store files (MinIO first) - #507
Draft
yifanxuaaa wants to merge 40 commits into
Draft
yifanxuaaa wants to merge 40 commits into
yifanxuaaa wants to merge 40 commits into
Conversation
Adds backend/sqlite/snapshot.rs: one consistent copy of a live Store to a new file through the SQLite online backup API (rusqlite's `backup` feature; no new crate). The copy runs on its own read-only connection in one backup step, so it reads one snapshot while this session and writers in every process continue under WAL. The destination must not exist; it is left closed in WAL mode with no sidecar and the source's schema version, ready for the sealed-Store install. A busy step is reported Busy, never retried; on failure an incomplete destination may remain. Public calls: Handles::snapshot, StorageProvider::snapshot, and OpenedStore::snapshot in layerfs-store-service (for push, session b). Test added: layerfs-persistence tests/snapshot.rs. No tests deleted or moved. Production LOC: 102831 -> 102950 (delta +119) Core 38395 -> 38395; engine 61777 -> 61896; api 2659 -> 2659; dsh 0 -> 0. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: tools/production_loc.py blob 3341e4d --root <snapshot> on exact parent 35d3657 and the committed tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#506 A1) Moves the SQLite engine into role folders; no behaviour change: - session/: connection, transaction, prepared, query, rows, profile, statement_work, thread_work - lane/: lane, admission, writer_admission - objects/: publish, read (was objects_read), selection (was objects_selection), unit_io, unit_layout, units_publish, units_read - metadata/: allocation, locations, policy, signatures, values (were metadata_*) - storage/: merge_records (from src/storage/) - acquisition/: + provider.rs (from src/storage/acquisition/) - lifecycle/: seal, seal_allocation, file_control, headroom, reclamation - schema.rs and snapshot.rs stay at backend/sqlite/ Deviation from the issue's list: src/history/ and src/storage/publication.rs stay shared. None of their files calls rusqlite or holds SQL text: history runs named statements through the backend transaction facade, and publication only validates a batch. The PostgreSQL engine reuses both. Submodule visibility widened to pub(crate) where a file moved into a different group from its callers (disable_persistent_wal pub(super) -> pub(crate)); include_str! paths gained one level. Public names (MergeScratch, MergeRecords, SqlWork, SqlitePersistenceProfile, AcquisitionProvider, ...) are unchanged. Tests: none deleted or moved. layerfs-persistence (all 37 test targets) and layerfs-store-service suites pass on macOS; clippy -D warnings clean. Production LOC: 102950 -> 102997 (delta +47) Core 38395 -> 38395; engine 61896 -> 61943; api 2659 -> 2659; dsh 0 -> 0. Relocation only: the +47 lines are the new role mod.rs declarations and grouped `use super::{..}` imports split across groups. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: tools/production_loc.py blob 3341e4d --root <snapshot> on exact parent 514a79f and the committed tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…506 R) New engine crate layerfs-remote-storage, which knows nothing about Stores: - driver.rs: Driver trait (limits, upload_file of a file or span, download_file whole or ranged, stat, delete). Each request is sent once; a sent request with an unknown outcome is Uncertain and never resent. - limits.rs (max file, max single request, piece size), error.rs, credentials.rs (env:LAYERFS_* or owner-only file reference; settings never hold the secret). - pieces.rs: a file over the provider's max file goes up as fixed-size pieces plus a manifest of each piece's size and sha256; the whole file's sha256 and layout ride as metadata. Download verifies every piece and the whole file. Pieces first, manifest last; stale pieces of an earlier, larger upload are removed. - local_folder/ (feature local_folder): folder driver for tests. - s3_api/ (feature s3_api): the retired one-attempt SigV4 client (da0fa3a^:core/crates/layerfs-s3) revived over hand-written HTTP/1.1, now one connection per request with streamed file and reply bodies, canonical query strings, and multipart upload (create, equal parts, complete; abort on any failure; 5 MiB-5 GiB parts, at most 10,000). Plain http:// only: TLS, needed for R2 and AWS, is a later phase. - minio/ (feature minio): limits 50 TiB object, 5 TiB single PUT, 64 MiB default parts; settings from LAYERFS_REMOTE_* or a credential reference. Tools: the crate joins the boundary guard's layer map (engine, no first-party dependencies, forbid(unsafe_code)) and production_loc.py's engine crate list. Cargo.lock gains the crate's entry (sha2 0.11.0, already locked); no new dependency. Tests added: tests/folder_pieces.rs (5), tests/minio.rs (1 default plus 6 ignored service tests run against a pinned MinIO via tests/minio_service.sh: single PUT/ranged GET/stat/delete, multipart above the part size, pieces, refused part aborted with no parts left, lost part reply Uncertain and not resent then aborted, failed abort reported PartsLeft and listed). No tests deleted or moved. Production LOC: 102997 -> 104696 (delta +1699) Core 38395 -> 38395; engine 61943 -> 63642 (layerfs-remote-storage 0 -> 1699); api 2659 -> 2659; dsh 0 -> 0. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: tools/production_loc.py blob 686d269 (this commit's counter, which adds the crate to the engine list; the parent has no such crate, so its total is unchanged) --root <git archive> of exact parent a9903e1 and the staged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…R) layerfs-store-service/src/remote/: - settings.rs: a project's named remotes in one text file the caller places; each names its provider (minio, local_folder) and a credential reference (env:LAYERFS_* or file:/path). Unknown keys, and so any secret, are refused; the file is written owner-only through a renamed sibling. - push.rs: one consistent snapshot of the live Store (OpenedStore::snapshot, session a's SQLite backup call; no SQLite code here) to a temporary file, its size checked against the remote's limits (pieces above max file, with the manifest kept within what a pull reads), uploaded with its sha256 and SQLite version as metadata. The temporary snapshot is removed. - pull.rs: download to `<store>.pulling` beside the local Store, verify the whole file's and each piece's sha256, open it as this project's Store (binding and cursor key), then rename it over the destination. Like install_local it needs the kernel guard and the daemons' user. An existing Store is replaced only when no opener holds it (no -wal/-shm sidecar); the caller stops openers first. A failure keeps the temporary file and names it. install.rs: installer_user is now pub(crate), shared with pull. No compare-and-swap on the remote (owner): a pull racing a push fails its sha256 check. Each request is sent once; nothing is resent. Tests added: layerfs-store-service tests/remote_settings.rs (2); layerfs-api tests/remote_store.rs (Linux): folder_round_trip and the ignored minio_round_trip push a Store with Branches and Commits, pull it into a fresh volume with identical Branches, Commits and files, check a later Commit is not in the snapshot, that an open Store is not replaced and a closed one is, and that another project's binding is refused. No tests deleted or moved. Production LOC: 104696 -> 105080 (delta +384) Core 38395 -> 38395; engine 63642 -> 64026 (layerfs-store-service 10140 -> 10524, layerfs-remote-storage 1699 -> 1699); api 2659 -> 2659; dsh 0 -> 0. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: tools/production_loc.py blob 686d269 --root <git archive> of exact parent 5586d8f and the staged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ser (#506 R) core/docs/architecture/86-remote-store-files.md describes the remote storage crate and the Store push/pull implemented in the two previous commits: one attempt per request, the pieces layout with its sha256 manifest, S3 multipart and abort, MinIO limits, named remotes with credential references, and the snapshot, verify and replace steps; plus what this phase leaves out (TLS, other providers, SDK methods, incremental push). Indexed in the architecture README. remote_store.rs: as local_install.rs, a non-root run (GitHub's runner) checks that pull is refused for a user other than the daemons' instead of failing. pull.rs: comment only (the sidecar rule holds on Linux, where a pull runs; macOS's system SQLite keeps empty sidecars). No tests deleted or moved. Production LOC: 105080 -> 105080 (delta 0) Core 38395; engine 64026; api 2659; dsh 0 (comment-only source change). Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: tools/production_loc.py blob 686d269 --root <git archive> of exact parent b55e317 and the staged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One Store holds many projects (named layer stacks) and a home holds many Stores: local SQLite Stores, PostgreSQL connections and remote sources. The doc records the owner direction of 2026-10-29, the model and what it changes (per-Store sync and writer lane, sharing within a Store, project lifecycle, addressing), the workflows (Init, add or remove a project, push, pull, clone, import), the live path on macOS, the planned code layout, the runtime calls integrations use, and the work split. Docs only; no code or tests change. Production LOC: 105080 -> 105080 (delta 0); core 38395, engine 64026, api 2659; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Counted with tools/production_loc.py --root on a git archive of the parent 9c6ebef; this commit adds no counted file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…space checks (#506 R) Design adjustment from the parent session (Connect's Storage page and the owner's Store-first direction): the unit of push and pull is a Store, with every project in it; a home has many Stores. - settings.rs: remotes are defined once per LayerFS home (`HOME/remotes`, Remotes::home_path). One remote holds many Stores: store_object(NAME) is `NAME/store.sqlite` (`<prefix>/NAME/store.sqlite` on MinIO), plus pieces. The fixed STORE_OBJECT is replaced by STORE_FILE and store_object. - state.rs (new): the last push or pull of each Store (remote, direction, time, sha256, size) in `HOME/remote-state/NAME`, outside the Store, since a pull replaces the Store file. Written owner-only through a renamed sibling. - push(opened, store, driver, name): the snapshot is staged beside the Store (`store.sqlite.pushing`, so inside the Docker volume on macOS) after checking room for one Store-sized copy (main file plus WAL) plus the Store's write headroom; PushFailure::Space names the directory, bytes needed and bytes free. Pushed carries the remote file name. - pull(driver, name, destination, ...): refuses an open Store before the download as well as before the rename, with an error naming the openers (daemon, mounted Workspace, SDK or worker process) and the macOS worker's quiet minute. The download stays a sibling of the Store, so the swap is one rename. - s3_api/config.rs: an endpoint may be any host name (container names with `_`), IPv4 or bracketed IPv6, with an optional trailing `/`; nothing assumes localhost. minio_service.sh also prints the in-network endpoint. Tests: remote_settings.rs gains Store names and the state file (3 tests); remote_store.rs pushes two Stores to one remote, records push and pull state, and checks an open Store is refused before any download; minio.rs checks the accepted and refused endpoints. No tests deleted or moved. Checked: macOS remote-storage minio (7, against MinIO) and folder_pieces (5), store-service remote_settings (3); Linux remote_store folder and MinIO round trips, the MinIO one reaching it by container name on a Docker network; Clippy -D warnings on Linux and macOS. The Space refusal has no automated test. Production LOC: 105080 -> 105282 (delta +202) Core 38395 -> 38395; engine 64026 -> 64228 (layerfs-store-service 10524 -> 10704, layerfs-remote-storage 1699 -> 1721); api 2659 -> 2659; dsh 0 -> 0. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: tools/production_loc.py blob 686d269 --root <git archive> of exact parent 69e2909 and the staged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and checks the remote copy (#506 R) For the Store-first API layer (session c), which calls push, pull and clone of a whole Store (core/docs/issues/506/README.md sections 3 and 6): - clone_store(driver, name, destination, cursor_key) -> Cloned: pull into a new Store. The destination must be absent; the binding comes from the remote copy and is returned; cursor_key is the new Store's own authority (open checks only the binding). The verified file is hard-linked into place, so a Store that appeared meanwhile is never replaced. - pull and clone share one fetch; pull now refuses, before any download, a remote copy whose recorded binding is another Store's. - push(opened, store, driver, name, binding, expected): the binding (not a secret) rides as `layerfs-binding` metadata for clone; Expected (Any, Absent, Sha256 of this home's last transfer) must match the remote copy, checked before the snapshot and just before the upload, else PushFailure::RemoteMoved. Not atomic: the owner requires no compare-and-swap, so a push by another machine in between is not seen. Tests: remote_store.rs clones into a new volume with identical history, refuses a clone over an existing Store, refuses pushes whose remote copy moved (Sha256 mismatch, Absent when present) without uploading, and refuses a pull of another binding before downloading. No tests deleted or moved. Checked: Linux remote_store folder and MinIO round trips (MinIO by container name on a Docker network); macOS remote_settings (3); Clippy -D warnings on Linux and macOS. Production LOC: 105282 -> 105436 (delta +154) Core 38395 -> 38395; engine 64228 -> 64382 (layerfs-store-service 10704 -> 10858, layerfs-remote-storage 1721 -> 1721); api 2659 -> 2659; dsh 0 -> 0. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: tools/production_loc.py blob 686d269 --root <git archive> of exact parent 6ad83a6 and the staged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds backend/postgresql/ behind a new `postgresql` Cargo feature (off by default), mirroring the SQLite role folders: - session/: connection with TLS (native-tls; sslmode from the URL), prepared-statement cache, error mapping by SQLSTATE (lost replies and closed connections are Unknown and quarantine the session), and profile.rs: schema, `synchronous_commit` (default off) and lane lock_timeout, all applied with SET LOCAL in the BEGIN round trip. - schema.rs + sql/postgresql/schema.sql: the SQLite logical tables in a configurable schema; bigint/bytea; names COLLATE "C" (SQLite byte order for paging); pack bodies bytea with STORAGE EXTERNAL; plus `store_lane`. - lane/lane.rs: one READ COMMITTED transaction on its own connection holding the lane row FOR UPDATE under SET LOCAL lock_timeout. Every write transaction takes the same row first (unless nested in this thread's held lane), so writers of every process run one at a time as on SQLite. Never SERIALIZABLE, never retried; a timed-out wait is Busy. - objects/: COPY BINARY of pack bodies in a write that does not wait for the lane, then locations/values/signatures/totals in the lane transaction; whole and ranged (`substring`) pack reads. - metadata/, history/ (two SQLite-only statements replaced; the rest of the shipped history SQL is portable and runs unchanged), storage/ (PostgresStorage: PackPersistence). The seam: backend/mod.rs gains `Mechanics` and `HistoryStatements`; shared metadata/objects bounds are generic over `Mechanics`; history's facade wraps `&dyn HistoryStatements`; HistoryProvider holds an `Engine` enum (SQLite or PostgreSQL). Admission (the FIFO turn queue) moves from backend/sqlite/lane/ to backend/admission.rs, shared by both engines; the pure pack-descriptor check moves to backend/records.rs. `BackendSelection::Postgres` now carries a `PostgresSelection` (URL, schema, synchronous_commit, lock_timeout_ms; Debug redacts the URL) and is no longer Copy; `Handles` still refuses it (Init stays SQLite only). New public API (feature): PostgresHandles (open_writable, open_read_only, close, diagnostics), PostgresStorage, PostgresLaneHold. Open path: layerfs-store-service gains a forwarding `postgresql` feature; open_store_observed opens a PostgreSQL selection with one writer session and one read-only session per Workspace reader (merges refused: no merge records on PostgreSQL yet); StoreService::open_postgres skips the SQLite kernel guard. layerfs-bridge ProviderKind::Postgres (wire code 2) is reserved and refused by install manifests (no sealed file). The import (A4) and the identical-ids test land in the next commit. New dependencies (owner-authorized for #506), all optional behind the feature: postgres =0.19.14 (pinned as in #420), postgres-native-tls =0.5.3, native-tls =0.2.18 in layerfs-persistence; postgres =0.19.14 in layerfs-store-service (its feature's tests). Cargo.lock adds 64 packages (tokio, tokio-postgres, postgres-protocol, native-tls/openssl/security- framework, ...) and moves js-sys 0.3.104->0.3.106 and wasm-bindgen* 0.2.127->0.2.129, required by whoami (wasm targets only). Tests: layerfs-bridge install_records gains a_postgresql_provider_is_never_an_install_manifest; sqlite_publication::wal_full_settings_and_unavailable_engine_are_explicit now builds the data-carrying Postgres selection (same assertion). No test deleted or moved. Checked on this tree (macOS): fmt; clippy -D warnings on persistence and bridge with and without the feature; workspace check; store-service check with the feature; persistence + bridge tests 145 passed, 0 failed. Production LOC: 105436 -> 107114 (delta +1678) Core 38395 -> 38395; engine 64382 -> 66060; api 2659 -> 2659; dsh 0 -> 0. layerfs-persistence 8580 -> 10158 (+1578): new PostgreSQL engine 1420 (backend/postgresql, sql/postgresql, store/postgresql.rs); shared seam, config and moves +158. Store-service and bridge make up the rest. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: tools/production_loc.py blob 686d269 --root <snapshot> on exact parent 7247d27 and the committed tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ds (#506 A4) Adds backend/postgresql/import.rs and PostgresHandles::import(source, target, binding, cursor_key) -> ImportReceipt. One SQLite read snapshot is copied into a new schema (refused if it exists) in one PostgreSQL transaction: the schema exists with every row or not at all. Rows move in bounded pages (4096 rows or 64 MiB of pack bodies) by COPY BINARY. Pack bodies are read through the SQLite engine's own pack read, so every pack layout imports. Ids, roots, Commits and inode serials are copied unchanged; the engine-scoped schema version and recorded schema become PostgreSQL's; totals are written last (the logical-size trigger runs on copied Layers and Commits). Init stays SQLite only. The source opens writable and is only read: a read-only SQLite connection cannot open a closed WAL Store without its -shm sidecar (a sealed or snapshot file). Existing behaviour of Handles::open_read_only, recorded on #506. Also adds Transaction::batch and schema::record, used only by import. Tests added (run against postgres:17-bookworm pinned by digest sha256:639ab7ceb90e13123085b741fb31ef493fba25463002f6da665352e7b534b652, via LAYERFS_TEST_POSTGRES_URL; NOT_RUN without it): - layerfs-persistence tests/postgresql_identical_ids.rs: one_project_has_identical_ids_on_sqlite_and_postgresql (SQLite project, live snapshot, import, the same Saves and Commits on both engines: identical object, root and Commit ids; reads from a fresh read-only session; existing schema and wrong binding refused) and a_held_lane_makes_another_writer_wait_then_fail_busy_once. - layerfs-store-service tests/postgresql_open.rs: a_postgresql_store_opens_through_the_store_open_path. No test deleted or moved. Persistence suite with the feature, all 3 new tests passing against Docker PostgreSQL 17.11 (macOS host). Production LOC: 107114 -> 107544 (delta +430) Core 38395 -> 38395; engine 66060 -> 66490; api 2659 -> 2659; dsh 0 -> 0. PostgreSQL engine 1420 -> 1850. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: tools/production_loc.py blob 686d269 --root <snapshot> on exact parent c94749e and the committed tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ath (#506) tools/check_product_boundary.py reported six violations from #506 A1/A2: - The audited macOS file-control module moved to backend/sqlite/lifecycle/file_control.rs in A1. The guard's allowed path, its self-test paths and engine/AGENTS.md now name the new path; AGENTS.md also records the second caller (a snapshot's own new destination connection), the same exclusively owned use. - backend/mod.rs held the Engine enum and the two seam traits; they move to backend/seam.rs and mod.rs re-exports them. - backend/postgresql/history/mod.rs held the statement table; it moves to history/statements.rs. No behaviour change. Checks: guard PASS (1013 files); guard self-tests 34 OK; ruff 0.17.2 format --check on both tools; clippy -D warnings on layerfs-persistence with and without `postgresql`; persistence suite with the feature against Docker PostgreSQL: 121 passed, 0 failed. Tests: test_check_product_boundary paths updated (same assertions); none deleted or moved. Production LOC: 107544 -> 107547 (delta +3) Core 38395 -> 38395; engine 66490 -> 66493; api 2659 -> 2659; dsh 0 -> 0. Relocation: two new files' declarations and re-export lines. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: tools/production_loc.py blob 686d269 --root <snapshot> on exact parent ee7cd91 and the committed tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#506 c) One Store holds many projects (owner, 2026-10-29). layerfs-project gains store/live.rs: - ProjectApi::create_store: a new empty Store, created with the acquisition tables and sealed, so projects can be added later. - ProjectApi::add_project: runs the existing namespace Init into an already-open Store over a writable session. Its Saves and history writes go through that Store's admission and writer lane, one short job at a time; the Store is never sealed or replaced. A taken name is refused (InitError::Exists) before anything is written. Init statistics stay the first Init's (recorded once per Store). - A Store without the acquisition tables (schema 16-18) is refused at the first acquisition unit, before any scan: InitError::NoAcquisition. No tables are added to an existing Store. Another add still working in the same Store is InitError::Busy. - store::projects: a Store's stacks, paged, in name order. Removing a project (dropping its stack) needs a stack removal in layerfs-persistence (session a); its pack reclamation is A8/#411. Test: layerfs-api/tests/store_projects.rs (an empty Store takes two projects and refuses a taken name; an Init-made Store takes a second project; a Store without acquisition tables is refused). No tests deleted or moved. Production LOC: 107547 -> 107723 (delta +176) Core 38395 -> 38395; engine 66493 -> 66669 (layerfs-project 2105 -> 2281, layerfs-api 1940 -> 1940); api 2659 -> 2659. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: tools/production_loc.py blob 686d269 --root <git archive> of exact parent f84f1d2 and the staged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fast CI lane's `fmt --check` reported import order and one long chain in tests/postgresql_open.rs. Formatting only; no test deleted or moved. Production LOC: 107547 -> 107547 (delta 0). Scope and method as in c94749e (tools/production_loc.py; tests/ outside src are not counted). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… schema's binding (#506) For session c's Store-first API (project removal, attaching a PostgreSQL Store in another home): - layerfs-history: `HistoryCatalog::remove_layerstack(stack)`, a default method that refuses `Unsupported`, so the test fakes are unchanged. - layerfs-persistence implements it on both engines (src/history/removal.rs and nine portable statements under sql/sqlite/queries/history/removal_*): - In one write transaction it deletes the stack's Layers, Branches and Commits, their layer/commit lineage and `absorbed`/`reach` rows, and subtracts their logical bytes from `store_totals`. - Packs, objects, file digests and the stack's allocator scope stay; the kept scope stops its serials from being reused. Reclaiming packs is #411. - An unknown stack is `Missing(LayerStack)`. - The write holds the writer for time that grows with the removed project's history; noted on #506. - `PostgresHandles::binding(selection)` reads the binding key a LayerFS schema records, in one read-only session after the table-set check that opening also runs (now one shared `schema::tables`). - README: both calls. Test added: tests/layerstack_removal.rs. On SQLite, and on PostgreSQL 17.11 (pinned digest) when LAYERFS_TEST_POSTGRES_URL is set, it removes one of two projects and checks: - the other stays whole and keeps committing; - the name can be reused with fresh lineage; - the totals return to the one-project value; - a second removal is Missing; - `binding` reads the imported schema's key and refuses `public`. No test deleted or moved. Checked (macOS): - persistence suite with `postgresql` and the server: 123 passed, 0 failed; - clippy -D warnings on persistence and history, with and without the feature; - cargo check --workspace --all-targets; - fmt --all --check; - boundary guard PASS. Production LOC: 107723 -> 107808 (delta +85). Scope and method: tools/production_loc.py over the parent and committed trees (core+engine product src and shipped SQL; tests, docs and tools excluded). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…c, PostgreSQL (#506 c) The native binding's backend calls for the Store-first model (doc section 6): - stores.rs: Store records in the home (`.stores/<store>/store.record`, project entries under `projects/`; a Store made by its project's own Init keeps its old entry), `runtime.store.create` (a new empty Store, under the kernel guard, renamed into place), `list_projects`, `add_project` (Init into the live Store through its writer lane and admission: Exists, Busy, Unsupported without acquisition tables) and `remove_project` (drops the stack with HistoryCatalog::remove_layerstack; packs stay until A8/#411; the founding project's entry is replaced by a Store record first), `forget`. - sync.rs: `runtime.remote.list/add/remove` (credential references only), `runtime.store.push` (expected copy, Conflict when the remote moved, `force`), `pull` (Busy while this process holds the Store), `clone`, `get_copy`; per-call MinIO keys; transfers recorded with session b's state. - postgres.rs (feature `postgresql`; Unsupported without it): open, `import_postgresql` (every project, the same ids) and `attach_postgresql` (a schema this home imported, or any LayerFS schema by the binding it records, with a fresh cursor key). URLs are passed per call, never stored. - direct.rs, discovery.rs, sandbox.rs: open by Store and project; a shared Store's Sandboxes are `<prefix><store>.<project>-<n>`, labelled layerfs.store; a PostgreSQL Store has no Sandbox. - Cargo feature `postgresql` forwards to persistence and store-service. Checks: clippy -D warnings with and without the feature (macOS); the real-binding E2E api/runtime/test/verify/stores_real.mjs passes on Linux in-process and through the macOS worker (next commit). Tests deleted: none. Production LOC: 107808 -> 109015 (delta +1207) Scope: core, engine and api production code, tools/production_loc.py (blob 686d269) at the parent and the staged tree; layerfs-node 2744 -> 3951. reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s, sync jobs, PostgreSQL (#506 c) @layerfs/runtime gains the doc section 6 calls, every one made once: - runtime.store.list/create/remove: `<home>/stores.json` lists the Stores (no secret); a local SQLite Store is one volume `<prefix>store-<name>` (labels layerfs.role=store, layerfs.home, layerfs.store). A Store made by its project's own Init is listed from that project's record (`legacy`), so existing projects still list and open. - runtime.project.start_init(name, folder, { store }) adds a project to a live Store (on macOS in its own container, as Init is); runtime.project.remove(name, { store, confirm, stopSandboxes }) drops a project's stack from a Store that holds others, after its Sandboxes (Busy while one runs); a project alone in its own Store is removed whole. list, get_state and sandbox.reclaim take `{ store }`. - runtime.remote.list/add/remove; runtime.store.start_push (force), start_pull (confirm; closes this process's clients first), start_clone ({ store, as }), start_import_postgresql, attach_postgresql; runtime.sync.get(id). Keys and PostgreSQL URLs pass per call only. - The worker forwards the new calls; a pull first closes the worker's hold on the Store. - @layerfs/agent opens (store, project): LAYERFS_STORE, and LAYERFS_STORE_POSTGRESQL for a PostgreSQL Store. - core/docs/issues/506/README.md section 6: the built names and shapes (added `{ store }` on get_state/reclaim, start_clone `{ store, as }`, import `{ name, connection, url, … }`, push `force`, remove results), the home layout, and the worker build command. Tests: runtime.mjs `stores` (every new call, against the fake binding); verify/stores_real.mjs with the real binding, MinIO and PostgreSQL 17: one Store holds two projects beside a second Store; a Sandbox commits; push, clone with identical history, pull over a later Commit; import with the same Branch and Commit ids, read through @layerfs/agent; attach by record and by schema; remove one project and the founding project. It passes on Linux in-process (`linux`, in the runtime image, with a test-only docker CLI shim) and through the macOS worker (`macos`). All 15 runtime cases and the agent tests pass. Tests deleted: none. Production LOC: 109015 -> 109420 (delta +405) Scope: core, engine and api production code, tools/production_loc.py (blob 686d269) at the parent and the staged tree; @layerfs/agent 849 -> 864, @layerfs/runtime 1135 -> 1525. reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`tools/worktree.py build --features <list>` passes the features to the build's cargo command, which the bundle manifest records. The worker's Linux build for PostgreSQL Stores is `--platform linux --features layerfs-node/postgresql`; MinIO is on by default. docs/general/parallel-worktrees.md says so. Test added: test_build_features_reach_cargo_and_the_manifest (54 tool tests OK, Ruff format and check clean). Tests deleted: none. Production LOC: 109420 -> 109420 (delta 0) Scope: core, engine and api production code, tools/production_loc.py (blob 686d269); tooling and docs only. reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…506 c) runtime.project.remove(name, { store, confirm, url }) on a PostgreSQL Store passes the URL for that call (InvalidInput without it); it was sent with none, so the binding could not open the schema. Doc section 6 names `url`. Tests: runtime.mjs `stores`; stores_real.mjs removes a project from the attached PostgreSQL Store, on Linux in-process and through the macOS worker. Tests deleted: none. Production LOC: 109420 -> 109422 (delta +2) Scope: core, engine and api production code, tools/production_loc.py (blob 686d269) at the parent and the staged tree; @layerfs/runtime 1525 -> 1527. reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Matching the coordinating session's Connect contract:
- runtime.store.list `copies` is one entry per remote,
`{ remote, pushed: { at, bytes, sha256 } | null, pulled: … | null }`,
instead of the latest transfer alone. The binding keeps recording each
transfer with remote::state (HOME/remote-state/<store>).
- runtime.store.start_clone(remote, { from, store }) clones the remote
copy of `from` into the new local Store `store`; `{ store, as }` still
works.
- Doc section 6 says so, and that sync.get(id).error.message is readable
as is (a push's free-space refusal is code Capacity).
Tests: runtime.mjs `stores`; stores_real.mjs on Linux in-process and
through the macOS worker. Tests deleted: none.
Production LOC: 109422 -> 109423 (delta +1)
Scope: core, engine and api production code, tools/production_loc.py
(blob 686d269) at the parent and the staged tree; @layerfs/runtime
1527 -> 1528. reference absent (retired R9c; last counted 65,417 at tree
498dd19).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… R) The S3 API driver now speaks HTTPS: an https:// endpoint runs the same one-request exchange over TLS (native-tls, the crate already pinned for PostgreSQL; OpenSSL on Linux, the system's roots, host name checked). The handshake comes before the request, so a refused certificate is Unavailable with nothing sent. TLS is a `tls` feature (remote-storage, store-service, layerfs-node), off by default like `postgresql`; without it an https:// remote is refused Unsupported when it is added. - layerfs-remote-storage: cloudflare_r2/ provider. An R2 remote is an account id (32 lowercase hex) and optional jurisdiction (eu, fedramp), reached at https://ACCOUNT[.J].r2.cloudflarestorage.com, path-style, region auto. R2 limits: 4.995 TiB object, 4.995 GiB single PUT, multipart above the part size. - s3_api: config accepts https:// (default port 443); http.rs wraps the socket in TLS for one Connection used to write and read. - store-service remote settings: `provider r2` with account, jurisdiction, bucket, prefix, credentials, part-size. - layerfs-node: runtime.remote.add/list for r2; per-call keys for any S3 provider. - minio_service.sh: MINIO_TLS_DIR serves HTTPS with a fresh self-signed certificate and exports SSL_CERT_FILE. - Docs: 86-remote-store-files, #506 README (calls, layout, build feature). Tests: remote_settings.rs gains r2_remotes_round_trip_and_derive_their_ endpoint; minio.rs gains multipart_direct_round_trip (no proxy); the four fault-proxy tests read plain HTTP, so against an https:// MinIO they say they were skipped and return; part_sizes_and_config_refusals checks https:// both ways. No tests deleted. Checked: - Clippy -D warnings on macOS for remote-storage, store-service and layerfs-node, with and without tls. - remote_settings 4/4 on macOS with and without tls. - Linux, in the build image on a private Docker network: - minio.rs 8/8 over plain HTTP; - 8/8 against a TLS MinIO with --features tls, the four proxy tests skipped as described; - an untrusted certificate is refused at the handshake (Unavailable). - Sealed Linux bundle with layerfs-node/postgresql,tls; through the macOS worker an r2 remote is added with its derived endpoint, and a push with fake keys to a made-up account reaches Cloudflare, which ends the TLS handshake (alert 40), as curl does. NOT_RUN: a real R2 account (needs an account id and keys). Production LOC: 109423 -> 109626 (delta +203) Scope: core, engine and api production code, tools/production_loc.py (blob 686d269) on git archives of the parent (1ce6711) and the staged tree; core 38400 -> 38400, engine 67956 -> 68159 (layerfs-remote-storage 1721 -> 1851, layerfs-store-service 10952 -> 10985, layerfs-node 3951 -> 3991), api 3067 -> 3067. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Sandbox's daemon now serves a project whose Store is a PostgreSQL schema: Workspaces mount, Exec runs and Commits publish into the schema, as on SQLite. Plan: #506 (comment) - bridge: DaemonSetup gains `postgresql: Option<DaemonPostgres>` (URL, schema, synchronous_commit, binding, cursor key), exclusive with existing_store; config magic LFSD\x06. The URL lives only in the daemon's private 0600 config file; Debug redacts it. - daemon: opens the selection through the same open_store_observed and starts ControlReady with no install and no Store file checks; the status names the engine `postgresql/<version>` (one printable word, as the status wire requires). New `postgresql` feature forwards to layerfs-store-service/postgresql. - sandbox: SandboxRequest.store_volume is Option; None mounts nothing and reopen expects zero mounts. - api: SandboxApi::create_postgres; create waits for the Store when the setup opens an existing one of either kind. - node: sandbox.create on a PostgreSQL Store reads the Store record and the call's postgresql.url, checks the project is in the schema, makes its home entry and starts a volume-less Sandbox. A Sandbox record may hold no volume. - agent: sandbox.create needs no LAYERFS_VOLUME on a PostgreSQL Store. - runtime: get_state and sandbox.reclaim take { url } for a PostgreSQL project instead of refusing; its client is opened for the call and closed after it. - docs: #506 section 6 and 8. Tests: daemon_setup_records (new), engine_protocol no-volume case, runtime `stores` stand-in case, and stores_real.mjs gains a PostgreSQL Sandbox step (mount, exec, Commit, read back; no Store mount, URL out of a command's reach). No tests deleted. Checks: stores_real.mjs linux and macos MAIN ok (bundle build-linux-d20dfdc3a53f4071ab7bbcfbed1405da, features layerfs-node/postgresql,layerfs-daemon/postgresql); Linux Clippy -D warnings on bridge, sandbox, daemon, api, node with both features; bridge, sandbox, node and api sandbox_operations host tests; runtime cases and agent calls. Production LOC: 109,626 -> 109,881 (delta +255): core 38,400 -> 38,400, engine 68,159 -> 68,388 (+229), api 3,067 -> 3,093 (+26); reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: tools/production_loc.py on `git archive` of the parent 809c86f and of the staged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Commit whose Branch moved since mount, and branch.merge, failed on a PostgreSQL Store with "pack persistence: BackendUnavailable": the Store had no merge records. Connect's tour scenario stopped there (its stale-Workspace step). - persistence: merge records keep their shared logic (resident records, write-back buffer, owner/token guards, windows) and reach their engine through `Scratch`: the SQLite Store file's merge_* tables as before, or, on PostgreSQL, temporary tables of the writer's connection (backend/postgresql/merge_records.rs). Those are outside the Store schema, which open checks exactly, so imported schemas open unchanged; they go with the connection, so no lease sweep is needed; writes run as Stage transactions and never take the publication lane. - store-service: a PostgreSQL Store opens with its merge scratch. - docs: #506 section 6 and 8. Tests: postgresql_open gains a merge-records case past the 8 MiB resident bound (guarded writes, stored reads, key windows, deletions, removal). Unchanged SQLite tests pass: persistence merge_records, old_format_open, lineage_backfill; api lane_merge. Linux Clippy -D warnings clean on persistence, store-service and daemon with postgresql. Connect scenarios on a PostgreSQL project: tour (with the stale-Workspace merge) passes. Production LOC: 109,881 -> 110,184 (delta +303): core 38,400 -> 38,400, engine 68,388 -> 68,691 (+303), api 3,093 -> 3,093; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: tools/production_loc.py on `git archive` of the parent 757b91b and of the staged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…506 d) Connect's depot and crew scenarios ran past their 15-minute limit on a PostgreSQL Store. pg_stat_statements on a copy of that Store showed the time was round trips, about 0.75 ms each over Docker Desktop's host.docker.internal: one statement per row where SQLite runs per-row statements in process. - Binding hints and file digests go as array (unnest) statements on PostgreSQL: up to 1024 rows per statement, and one statement per 64-content digest window. A repeated digest keeps the later root, as one row at a time would. SQLite keeps its per-row statements (the history seam's new `history_arrays` returns None there). - Immutable reads (locations, inode values, packs) run without BEGIN/COMMIT, as on SQLite. Their statements name the schema, so they do not depend on search_path. - A pack read fetches its descriptor and the body's first 64 KiB in one statement; header and directory reads inside that head cost no round trip. Whole-pack reads take every checked body in one statement. In the depot run, fs.apply went from 95.8 s worst case (9 calls, 108 s) to 0.4 s; one Commit's statements went from 39,555 to about 3,300. Tests: new hints_and_digests_in_arrays_read_back_as_on_sqlite (postgresql_identical_ids.rs) with 1,510 hints, 1,100 digests across the 1024-row boundary and a missing root; existing PostgreSQL tests pass (PostgreSQL 17); persistence crate tests pass; Linux Clippy -D warnings clean for persistence and store-service with postgresql. No test deleted. Production LOC: 110,184 -> 110,381 (delta +197), tools/production_loc.py on git archive of the parent and of the staged tree. core 38,400 -> 38,400; engine 68,691 -> 68,888 (+197, layerfs-persistence 10,971 -> 11,168, including 3 new SQL statements); api 3,093 -> 3,093. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… d) Doc section 6 now says how the engine keeps server statements few (autocommit immutable reads, a pack's head with its descriptor, array statements for hints and digests). It also says a server in a container on the same Docker engine is best reached at its bridge address: about 0.13 ms per round trip there against about 0.75 ms through host.docker.internal on Docker Desktop. Production LOC: 110,381 -> 110,381 (delta 0), tools/production_loc.py; docs only. core 38,400; engine 68,888; api 3,093. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Section D, continued: 🤖 Generated with Claude Code |
…rage (#506 E2 E4 E5 E9) Remote storage: a progress cell and thread-scoped reporting (hash, upload and download bytes, pieces; bytes published at most every 250 ms), Stat gains `modified` (Last-Modified), `Driver::list(prefix)` (ListObjectsV2 paging; folder walk), and uploads record the whole file's size as `layerfs-bytes` so one HEAD tells it even for a pieced file. Persistence: `read_heads(path)` reads the history heads lines of a Store file on a connection of its own (query_only, closed before it returns); the online-backup snapshot runs in steps that share one read transaction and reports bytes copied (`snapshot_observed`); `postgresql_status` reads a server's version, one round trip's latency and schema sizes on one connection. Also silences a dead-code lint on `Column` without the postgresql feature (pre-existing). Tests: remote-storage progress_listing (cell throttle, phases, listing, timestamps), minio listing_and_last_modified (ignored, needs MinIO); persistence heads. Production LOC: 110,381 -> 110,916 (delta +535), tools/production_loc.py on git archive of the parent and of the staged tree. core 38,400 -> 38,400; engine 68,888 -> 69,423 (+535: layerfs-persistence 11,168 -> 11,337, layerfs-remote-storage 1,851 -> 2,207, layerfs-store-service 10,988 -> 10,998); api 3,093 -> 3,093. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ostgreSQL status (#506 E1-E9) Store service: push records the snapshot's history heads digest and sends it as `layerfs-heads`; pull and clone read it from the downloaded file before installing; remote state keeps `heads` (older records read as none). `remote::check` is one stat of a Store copy (size, sha256, heads, modified); `remote::list_stores` lists every `NAME/store.sqlite` in one paged listing with at most 64 stats where the listing lacks metadata. Push and pull report check/snapshot/hash/upload and check/download/verify/install to the thread's progress. Binding (backendCall): runtime.store.stat (file and WAL sizes, open, heads; heads re-read only when the files changed, on a connection of its own, serialized with this process's pull and clone), runtime.store.check, runtime.remote.list_stores, runtime.remote.providers, runtime.postgresql.status and runtime.sync.progress; push, pull, clone and import report to the cell of the call's `job`, and transfers carry `heads`. Tests: store-service remote_copies; remote_settings gains the heads round trip and the pre-E2 record. Production LOC: 110,916 -> 111,484 (delta +568), tools/production_loc.py on git archive of the parent and of the staged tree. core 38,400 -> 38,400; engine 69,423 -> 69,991 (+568: layerfs-node 4,069 -> 4,486, layerfs-store-service 10,998 -> 11,149); api 3,093 -> 3,093. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… PostgreSQL status (#506 E1-E9) @layerfs/runtime, on both routes (Linux in-process, the macOS and Windows worker): store.list adds bytes { file, wal }, open and heads to each local SQLite Store (one runtime.store.stat call; nulls when it fails) and importedFrom; copies' pushed/pulled carry heads. New calls: runtime.store.check(store, { remote, credentials }), runtime.remote.list_stores(remote, { credentials }), runtime.remote.providers(), runtime.postgresql.status({ url, schemas }). Jobs pass their id to the native call; runtime.sync.get(id) adds phase, bytes/total and pieces of a running job from its progress cell. Imports record importedFrom { store, at, bytes } in stores.json. Tests: runtime.mjs `stores` covers every new call and field on the stand-in binding (fake-native gains the new ops and a push hold). Production LOC: 111,484 -> 111,525 (delta +41), tools/production_loc.py on git archive of the parent and of the staged tree. core 38,400 -> 38,400; engine 69,991 -> 69,991; api 3,093 -> 3,134 (+41, @layerfs/runtime 1,553 -> 1,594). Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…calls stores_real.mjs adds: store.list sizes, open state and heads; heads that move with a fork and Commit, equal the pushed copy's and come back with a pull; store.check against MinIO (present, absent, Denied on a wrong secret); remote.list_stores; remote.providers; push progress phases sampled while a ~96 MiB Store goes up; importedFrom; postgresql.status (version, latency, schema sizes, a schema the server lacks, Denied). Passed once on macOS through the worker and once on the Linux in-process route, bundle build-linux-a26402aa76354e0883832361c02db8c2 (postgresql). README section 6 gains the new calls and fields and the heads digest. Production LOC: 111,525 -> 111,525 (delta 0), tools/production_loc.py on git archive of the parent and of the staged tree; tests and docs only. core 38,400; engine 69,991; api 3,134. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Production LOC: 111,525 -> 111,525 (delta 0), tools/production_loc.py; test formatting only. core 38,400; engine 69,991; api 3,134. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, named read errors (#508) Sandbox control calls hung behind Workspace storage scans (#508): every runtime.project.get_state asked each Workspace for get_storage, which walked the whole base tree to count changed paths; abandoned polls kept the binding's connections busy, and a call waiting for one waited for ever while holding its Sandbox's discovery turn. A. backend.daemon.workspace.get_storage reads the overlay's row counts only and returns `changes: null`, like `metadata`. No caller read the counts (LayerFS api, Connect, dsh-layerfs). list_changes still names the changed paths on request; it still walks the base tree. B. Controls::lend waits at most LEND_WAIT (60 s, the admission bound) for a free connection, then refuses Busy: nothing was attempted, and the refusal is not retried. C. Object demand failures keep their kind: PortError::object_demand names concurrent demand, read admission, poison, history and storage failures instead of one "Store object demand" for all. Both sites (store ports, read cache) use it. Not changed (owner decisions in #508): whether a read-only call's uncertain failure stays retained by the Workspace's ReadSession (C), release of retained daemon connection slots (D), daemon-side cancellation of abandoned calls. Tests added: - layerfs-daemon mounted_storage_reads (Linux, FUSE): 10,000-file base, one added and one modified file; get_storage demands no Store object, list_changes lists both. Passed, 3.2 s. - layerfs-node native_connection_wait (Linux, Node 24 runtime image): a call behind a held connection is refused Busy after 60.0 s. Passed, 66.1 s; added to SLOW in tools/ci_tests.py. - layerfs-store-service object_demand_errors (host). Passed. Deleted tests: none. Checks: Clippy -D warnings on layerfs-store-service, layerfs-daemon and layerfs-node, all targets, macOS and Linux; rustfmt clean; tools/test_ci_tests.py passed. layerfs-node native_admission fails on this branch before reaching the changed code: discovery now inspects the container three times and the test's mock Engine serves two (both cases end with an Engine ConnectionRefused in 0.1 s). Not fixed here. Production LOC: 111,525 -> 111,540 (delta +15), tools/production_loc.py on the first-parent and committed trees. Engine 69,991 -> 70,006 (+15); core 38,400 and api 3,134 unchanged; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since #457 each Docker client first sends an unversioned GET /version. The native binding tests' mock Engine served that request one of its scripted container responses, so the inspection that followed found no listener: native_admission failed in 0.1 s with an Engine ConnectionRefused before any daemon connection, on main as on this branch. The mock now answers every GET /version with ApiVersion 1.54 and neither consumes a scripted response nor returns its head, so each test's scripted list stays the Engine calls it is about. native_connection_wait (#508) goes back to three scripted inspections. Linux runs in the Node 24 runtime image, one container per target: native_discovery 6 passed (0.5 s), native_admission 2 passed (60.2 s), native_connection_wait 1 passed (66.1 s). Clippy -D warnings on layerfs-node, all targets, Linux. Deleted tests: none. Production LOC: 111,540 -> 111,540 (delta 0), tools/production_loc.py on the first-parent and committed trees; test-only change. Engine 70,006, core 38,400 and api 3,134 unchanged; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 29, 2026
Owner direction 2026-10-30: each local SQLite Store links to at most one remote file, and a remote file is linked by at most one Store of the home. Connections bind to providers only. The spec replaces the per-remote copies model of README section 6 for SQLite Stores, names the new runtime calls (publish, send, load, unlink, push and pull by link), the native record's file line, tests, and the separate phase that settles today's data. Production LOC: 111,540 -> 111,540 (delta 0), tools/production_loc.py on the first-parent and committed trees; docs-only change. Engine 70,006, core 38,400 and api 3,134 unchanged; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner direction 2026-10-30 (links.md): a SQLite Store links to at most
one remote file, and a remote file is linked by at most one Store of the
home. The per-remote copies list disagreed with the native record, which
refused pushes wrongly ("the remote copy moved" against another remote's
sha256) and made every clone read missing.
Runtime (@layerfs/runtime):
- stores.json version 2: `link: { remote, file, linked, last } | null`
replaces `remote` and `copies`; version 1 is read with no link and the
first write keeps it as stores.v1.json.
- start_publish (unlinked Store to a new file, linked), start_send (a copy
to a new file, linked to nothing), start_load (replaces start_clone; a
new Store linked to the file), unlink; push, pull and check go through
the link and take no remote. list_stores items name `file` and
`linkedBy`; remote.remove refuses InUse while a Store links to it.
- One to one per home: a (remote, file) is claimed under the registry
lock (stores.json.lock, refused Busy, never waited for) before a job
that creates or links it, and linked under the lock when it ends.
Native:
- remote-state/<store> gains a `file` line and ms times; a record from
before reads as the Store's own file with its seconds as ms.
state::expected: a push expects what the link's own record names,
whatever its direction, and no file otherwise.
- binding ops runtime.store.publish/send/load/unlink/get_link; push,
pull and check use the object <prefix>/<file>/store.sqlite. Publish and
send HEAD the file first and before the upload: Exists, never
overwritten. remote.remove refuses InUse from the records; store.forget
drops the Store's record. clone_store/Cloned renamed load/Loaded.
- README section 6 rows and the section 3 diagram updated.
Tests: layerfs-store-service remote_settings (file line, ms, old records,
expected by link, remove) and remote_copies pass on the host (macOS);
runtime.mjs stores (publish, Linked/Exists/NotLinked, push and pull by
link, send, load, two loads of one file, InUse, unlink, v1 reading) and
every other runtime.mjs selection pass. layerfs-api remote_store gains
the links.md section 2 case (send, then push through the link passes;
publish onto an existing file refused). Clippy -D warnings on
layerfs-store-service and layerfs-node (macOS), fmt and the product
boundary guard pass. Deleted tests: none; the start_clone and copies
assertions of runtime.mjs stores and stores_real.mjs were replaced by
load and link ones.
Production LOC: 111,540 -> 111,799 (delta +259), tools/production_loc.py
on the first-parent and committed trees. Engine 70,006 -> 70,134
(+128: layerfs-node, layerfs-store-service), api 3,134 -> 3,265 (+131:
@layerfs/runtime), core 38,400 unchanged; reference absent (retired
R9c; last counted 65,417 at tree 498dd19).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6 tasks done
Owner direction 2026-10-30: a SQLite Store saved in S3 (MinIO, Cloudflare R2) is viewable without loading it: its folder tree, history and file contents, read-only. A temporary full download is rejected. The spec covers the VACUUM INTO push layout (blocked on a rowid audit), the read-only remote VFS with a bounded block cache and ETag pinning, the remote:<connection>/<file> store path, the runtime view calls, the Connect Browse view, verification and measurement. Production LOC: 111,799 -> 111,799 (delta 0), tools/production_loc.py on the first-parent and committed trees; docs-only change. Engine 70,134, core 38,400 and api 3,265 unchanged; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The push snapshot is written by VACUUM INTO on its own read-only connection (one read transaction, writers keep going), instead of the online backup API. Each table and index becomes one contiguous run of pages, so a ranged reader of the pushed file needs few requests: on the 251 MB r2-examples-clone diagnostic, history from 43 ranges to 3 and tree metadata from 259 to 2 (#510). Rowid audit (posted on #510): all 24 Store tables are WITHOUT ROWID or key their rows by an INTEGER PRIMARY KEY; no engine query relies on an implicit rowid; sqlite_sequence is copied. VACUUM INTO writes a rollback-journal file and refuses an existing output, so the copy is switched to WAL and its user_version, application_id, page_size and auto_vacuum are checked against the source before upload. The source connection runs synchronous=OFF, so the output is not synced (Disposable). Progress reports the copy's growing size every 100 ms, then its final size. The unused rusqlite backup feature is dropped. The snapshot test now also asserts the header fields, the WAL header bytes, no free pages and the progress report. Production LOC: 111,799 -> 111,799 (delta 0), tools/production_loc.py on the first-parent and staged trees (snapshot.rs 121 -> 121). Engine 70,134, core 38,400, api 3,265; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…510 B) A read-only, immutable SQLite VFS, `layerfs-remote`, reads a Store file on a remote through ranged requests, so a Store saved in S3 can be read without downloading it. - layerfs-persistence `remote/`: the VFS through rusqlite::ffi, the crate's only unsafe code (`remote/vfs.rs`, with safety comments; no new dependency). It opens main database files only, reports SQLITE_IOCAP_IMMUTABLE and opens them with immutable=1; xWrite, xTruncate, xSync and xDelete refuse SQLITE_READONLY; mmap is answered 0. `remote/cache.rs`: 64 KiB blocks, misses merged into ranged requests of at most 1 MiB (the last one reads ahead into the next missing blocks), a bounded LRU (default 64 MiB), counters; its lock is never held across a request. A failed read is kept on the file with its kind (Unavailable, Denied, Changed, Unknown, Failed); once Changed, misses fail without a request. `Handles::open_remote` opens one read-only Disposable session over it; the profile check skips the WAL journal check for this VFS only (an immutable connection reports none). - layerfs-remote-storage: `Stat::etag`; `Driver::download_range` with a `Condition` (If-Match, If-Unmodified-Since), which the S3 client and the local folder implement; the S3 client's GET is shared with download_file; HTTP 412 is the new `RemoteError::Changed`. `PinnedFile` stats a file once and reads that version only: a whole file under If-Match, a pieced file through its manifest, each piece pinned by the manifest's time on first read, then by its tag. One attempt per request; requests and bytes counted. - layerfs-store-service: `remote::open_view` (binding from the push metadata, a random per-view cursor key from the caller), `bootstrap::open_remote_store`, and `StoreService::read_only`, which has no overlay owner, so binding a Workspace is refused ReadOnly. Tests: persistence remote_view (Store bytes read in place equal the local file's, requests block aligned and at most 1 MiB, cache answers repeats, writes refused, wrong binding refused, Changed kept and no further request, block planning, cache bound); remote-storage folder_pinned (whole and pieced ranges, a range across pieces is two requests, replacement is Changed, conditions, HTTP dates); store-service remote_view (whole and pieced views open read-only, no binding or no Store refused, error kinds). Run on macOS (system SQLite); Linux runs at the native comparison. Production LOC: 111,799 -> 112,983 (delta +1,184), tools/production_loc.py on the first-parent and staged trees. Engine 70,134 -> 71,318; core 38,400 and api 3,265 unchanged; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
C: a view's store path, `remote:<connection>/<file>#<view>`, opens the
view's read-only service in place of a Store file (layerfs-node
`views.rs`, routed from `direct.rs`). Seventeen reads work on it
(project.list_branches, branch.list_commits/diff_commits/diff_branches,
fs.read/list/stat/grep/glob/diff_files/blame and the six
backend.store.* reads); every other call naming it, Workspaces and
Sandboxes included, is refused ReadOnly before it runs. A read whose
remote request failed answers that request's kind: Changed,
Unavailable, Denied, Unknown or Failed. A view is never in stores.json,
never linked, never a Store of the home.
D: binding calls runtime.remote.open_view (one HEAD pins the version,
then the reads that open the Store and list its projects; keys per
call, a random per-view cursor key, cache bound from
LAYERFS_REMOTE_VIEW_CACHE, default 64 MiB), close_view and view_stats;
views unused for a minute are closed at the next view call.
@layerfs/runtime: runtime.remote.open_view/close_view/view_stats,
runtime.project.list({ view }) and get_state(name, since, { view })
(history only, no Sandboxes); a view's projects read through agent
clients on its store path, closed with the view; views close after a
quiet minute and with the runtime. RemoteError::Changed answers
Changed in the binding's remote refusals. README section 6 records the
contract.
Tests: runtime `views` selection (open, list, state without Sandboxes,
the client opens the view's store path, NoSuchProject, stats, close
closes its clients, NoSuchView after); `stores` surface updated for
the three new calls; the fake native answers the view calls.
Production LOC: 112,983 -> 113,319 (delta +336), tools/production_loc.py
on the first-parent and staged trees. Engine 71,318 -> 71,565 (+247),
api 3,265 -> 3,354 (+89); core 38,400 unchanged; reference absent
(retired R9c; last counted 65,417 at tree 498dd19).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…510) A failed read quarantines the view's Store session, as on any Store, so a second read after Changed came back Unknown without a request. The binding now answers the view's remote failure (Changed, Unavailable, Denied, ...) on every later call that fails; the view is opened again to read on. sandbox.create on a view skips the deployment check, so the binding refuses it ReadOnly instead of InvalidInput (no image). Adds api/runtime/test/verify/view_real.mjs (#510 §7.3): a view of a Store in MinIO answers projects, Branch heads, Commits, listings, stats and whole files (small, chunked, large) as the local Store; writes, forks, Sandboxes and Workspaces are refused ReadOnly; a push mid-view gives Changed. Production LOC: 113,319 -> 113,319 (delta 0), tools/production_loc.py on the first-parent and staged trees. Engine 71,565 -> 71,564 (-1), api 3,354 -> 3,355 (+1); core 38,400 unchanged; reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runtime closed a view after a quiet minute unless a call ran through
views.use, so a host reading a view's projects through its own clients
(Connect) lost the view mid-page. Every lookup of an open view by a call
(project.list({ view }), get_state, view_stats) now starts its quiet
minute again.
Production LOC: 113,319 -> 113,320 (delta +1), tools/production_loc.py
on the first-parent and staged trees. api 3,355 -> 3,356 (+1); engine
71,564 and core 38,400 unchanged; reference absent (retired R9c; last
counted 65,417 at tree 498dd19).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Production LOC: 113,320 -> 113,320 (delta 0), tools/production_loc.py on the first-parent and staged trees; docs only. Reference absent (retired R9c; last counted 65,417 at tree 498dd19). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Shared PR for #506, worked by two sessions in one worktree. Areas A and R are done for this phase; area C (session c, Store-first API layer) is not in this PR yet, so it stays a draft.
A. Persistence (session a), complete for this phase
514a79fdcdconsistent snapshot of a live Store (SQLite backup API):Handles::snapshot,OpenedStore::snapshot.a9903e1286A1: SQLite code grouped by role underbackend/sqlite/, no behaviour change.c94749e591A2: PostgreSQL engine behind one seam (Mechanics,HistoryStatements), featurepostgresql.synchronous_commitis a setting, default off, applied withSET LOCAL.store_lanerowFOR UPDATEunderlock_timeout. A timeout isBusy; nothing is retried; a lost COMMIT reply isUnknown.substring.StoreService::open_postgresandProviderKind::Postgres, which install refuses.postgres =0.19.14,postgres-native-tls,native-tls.ee7cd91365A4:PostgresHandles::importcopies a SQLite Store into a new schema, keeping every id.f84f1d25f5boundary-guard fix: the guard now names the moved FFI pathbackend/sqlite/lifecycle/file_control.rs. It reports PASS.3092604cb0rustfmt of the PostgreSQL open test.740fd07615for session c:HistoryCatalog::remove_layerstack: one write transaction drops one project's rows and lineage on either engine.PostgresHandles::binding: reads the binding a LayerFS schema records.Deviations, findings and gaps are in the fix record on #506. The main ones:
Checks (session a, macOS, at
f84f1d25f5)3092604cb0fixes the one test file CI flagged atf84f1d25f5).-D warningson persistence, bridge and store-service, with and withoutpostgresql.Production LOC (A): 102831 → 102950 → 102997; 105436 → 107114 → 107544 → 107547 → 107547; 107723 → 107808.
R. Remote Store files, MinIO first (session b), complete for this phase
5586d8f6afnew crateengine/crates/layerfs-remote-storage:Driver(limits, upload a file or span, ranged download, stat, delete), one attempt per request (Uncertainis never resent), pieces + sha256 manifest above a provider's max file, folder driver, S3 API client revived fromda0fa3a120^:core/crates/layerfs-s3with streamed bodies and multipart (create/parts/complete, abort on failure), MinIO provider. Plain HTTP only; TLS (R2, AWS) comes later.b55e317c6elayerfs-store-service/src/remote/: named remotes with credential references (never the secret), push (snapshot → limits check → upload with sha256), pull (download to a temporary sibling → sha256 → opens as this project's Store → rename over the local Store only when no opener holds it).9c6ebef636architecture notecore/docs/architecture/86-remote-store-files.md; non-root handling in the Linux test.Checks (session b)
layerfs-remote-storagetestsfolder_pieces(5) andminio(7, including the 6 ignored service tests run against a pinned MinIOcgr.dev/chainguard/minio@sha256:4692462f…);layerfs-store-serviceremote_settings(2).layerfs-apiremote_store:folder_round_tripandminio_round_trippush a Store with Branches and Commits and pull it into a fresh volume with identical history.-D warningson Linux and macOS for the touched packages; rustfmt on changed files; boundary guard self-tests. (The guard violation from the persistence move is fixed inf84f1d25f5.)Production LOC per commit is in each commit message (R: 102997 → 104696 → 105080 → 105080).
C. Store-first API layer (session c), complete for this phase
At
60672a35e1: layerfs-project adds a project to a live Store and lists a Store's projects. The layerfs-node binding gains the Store-level calls (Stores, projects, remotes, push/pull/clone, PostgreSQL import/attach, project removal).@layerfs/runtimegains the doc section 6 calls (stores.json, one volume per Store, sync jobs), and@layerfs/agentopens (store, project).tools/worktree.py build --features layerfs-node/postgresqlbuilds the worker with the PostgreSQL engine. Details, deviations and LOC per commit: #506 (comment)Checks (session c)
api/runtime/test/verify/stores_real.mjs(real binding, MinIO, PostgreSQL 17) passes on Linux in-process and through the macOS worker. It covers two projects in one Store, a second Store, a Sandbox Commit, push and clone with identical history, pull, PostgreSQL import with the same ids, attach, and removals.store_projectspasses 3/3 on macOS and Linux.-D warningsis clean with and withoutpostgresql; the boundary guard passes; the tool tests and Ruff are clean.D. Sandboxes on PostgreSQL Stores (session d), complete for this phase
757b91b48e: a Sandbox's daemon serves a project of a PostgreSQL Store. Workspaces mount, Exec runs and Commits publish into the schema.get_stateandsandbox.reclaimtake{ url }per call.--features layerfs-node/postgresql,layerfs-daemon/postgresql.stores_real.mjslinux and macos: MAIN ok, with a new PostgreSQL Sandbox step.🤖 Generated with Claude Code