Single-writer storage model: durability fixes + topology guidance

README.md

@@ -12,21 +12,30 @@ Haverstack is a structured data store for individuals and small organizations. A
 
 A stack is a personal or organizational data store. It belongs to one **Entity** (a person or org) and holds **Records** — structured data objects that apps create and read.
 
-The key idea: apps only talk to the Haverstack library. They don't know or care whether the underlying storage is a local SQLite database, a folder of JSON files, or a remote server. Switch backends without changing your app.
+The key idea: apps talk to the Haverstack library, not to a storage format directly — switch backends without changing your app's data-access code. That said, storage ownership is exclusive: **a stack file is owned by exactly one process at a time.**
+
+---
+
+## How apps share a stack
+
+- **One app, one stack:** the app embeds `adapter-local` and owns the file directly. This is the simple, common case.
+- **Multiple apps, one stack:** run a server (local or hosted) that owns the file, and have each app connect through `adapter-api` — the same client works against `localhost` or a remote provider.
+
+Don't point more than one app at the same stack file with `adapter-local`. Nothing enforces permissions at that layer — `appId` is self-reported and grants aren't checked — so direct file access is a full-trust, single-owner arrangement, not a way to share data between apps. See [Concurrency & storage ownership](./docs/spec.md#concurrency--storage-ownership) in the spec for the full rationale.
 
 ---
 
 ## Packages
 
 This is a monorepo. Packages are published to npm under the `@haverstack` scope.
 
-| Package                                                               | Description                                           |
-| --------------------------------------------------------------------- | ----------------------------------------------------- |
-| [`@haverstack/core`](./packages/core)                                 | Stack class, types, schema, validation, ID generation |
-| [`@haverstack/adapter-local`](./packages/adapter-local)               | Local adapter (SQLite + disk) — the common case       |
-| [`@haverstack/record-adapter-sqljs`](./packages/record-adapter-sqljs) | sql.js (SQLite/WASM) `StackRecordAdapter`             |
-| [`@haverstack/blob-adapter-disk`](./packages/blob-adapter-disk)       | Disk filesystem `StackBlobAdapter`                    |
-| [`@haverstack/adapter-api`](./packages/adapter-api)                   | HTTP adapter for remote stack servers                 |
+| Package                                                               | Description                                                       |
+| --------------------------------------------------------------------- | ----------------------------------------------------------------- |
+| [`@haverstack/core`](./packages/core)                                 | Stack class, types, schema, validation, ID generation             |
+| [`@haverstack/adapter-local`](./packages/adapter-local)               | Local adapter (SQLite + disk) — single-app/embedded or server use |
+| [`@haverstack/record-adapter-sqljs`](./packages/record-adapter-sqljs) | sql.js (SQLite/WASM) `StackRecordAdapter`                         |
+| [`@haverstack/blob-adapter-disk`](./packages/blob-adapter-disk)       | Disk filesystem `StackBlobAdapter`                                |
+| [`@haverstack/adapter-api`](./packages/adapter-api)                   | HTTP adapter for remote stack servers                             |
 
 Planned:
 
@@ -148,13 +157,13 @@ The adapter interface is split into `StackRecordAdapter` (structured records) an
 - **`record-adapter-*`** — `StackRecordAdapter` only
 - **`blob-adapter-*`** — `StackBlobAdapter` only
 
-| Package                | Type   | Use case                                        |
-| ---------------------- | ------ | ----------------------------------------------- |
-| `adapter-local`        | full   | Local app storage — SQLite records + disk blobs |
-| `record-adapter-sqljs` | record | sql.js records, full query support, FTS         |
-| `blob-adapter-disk`    | blob   | Content-addressed blobs on the local filesystem |
-| `adapter-api`          | full   | Hosted/shared stacks via HTTP                   |
-| `adapter-json`         | full   | Portable JSON files _(planned)_                 |
+| Package                | Type   | Use case                                                        |
+| ---------------------- | ------ | --------------------------------------------------------------- |
+| `adapter-local`        | full   | Single-app/embedded or server use — SQLite records + disk blobs |
+| `record-adapter-sqljs` | record | sql.js records, full query support, FTS                         |
+| `blob-adapter-disk`    | blob   | Content-addressed blobs on the local filesystem                 |
+| `adapter-api`          | full   | Hosted/shared stacks via HTTP                                   |
+| `adapter-json`         | full   | Portable JSON files _(planned)_                                 |
 
 Use `combineAdapters({ record, blob })` from `@haverstack/core` to compose a record adapter with a different blob backend — for example, `SQLiteRecordAdapter` with a future `S3BlobAdapter`. `adapter-local` wraps this pattern for the common case.
 

docs/spec.md

@@ -483,6 +483,19 @@ All adapters support the full Record API. Performance guarantees differ; correct
 
 ---
 
+## Concurrency & storage ownership
+
+A stack's backing storage (a SQLite file, a JSON directory) has exactly one owning process at a time. Adapters are not required to support concurrent multi-process access, and the whole-file adapters (`record-adapter-sqljs` and the planned `adapter-json`) do not: they read the entire store into memory on open and rewrite it whole on every persist, so two processes opening the same store each hold an independent copy and silently clobber each other's writes.
+
+Multi-app access goes through a server implementation over the wire protocol (`adapter-api`, against `localhost` or a hosted provider) — never by pointing multiple apps at the same storage file directly. This is also the only topology in which permissions, grants, and `appId` attribution mean anything: `Stack` performs no permission checks, `ScopedStack` is opt-in, and `appId` is self-reported by the writing app, so direct storage access implies full trust over everything in the store, including grants and token hashes.
+
+Two properties adapters SHOULD provide regardless of topology, since even a single process can crash mid-write or be opened twice by accident:
+
+- **Fail loudly on double-open.** An adapter SHOULD detect that its storage is already open in another live process and reject `open()`/`initialize()` with a clear error, rather than silently entering a last-writer-wins mode. `record-adapter-sqljs` does this with a PID-stamped lock file beside the database, released on `close()`; a stale lock (owning process no longer alive) is reclaimed automatically, and an explicit override is available for the rare case of PID reuse.
+- **Persist atomically.** A crash mid-write must never leave storage unreadable. `record-adapter-sqljs` writes each snapshot to a temp file in the same directory and `rename()`s it over the target — atomic on POSIX, so there is no window where the file is truncated or partially written.
+
+---
+
 ## Attachments
 
 Binary files are stored and retrieved through the library using **content-addressed storage**. A file's ID is the SHA-256 hash of its bytes, so uploading identical bytes twice returns the same `fileId` without writing a second binary copy. Each upload creates a new `_attachment@1` record (see [Attachment](#attachment)) regardless of deduplication, so metadata (mimeType, size, filename) is tracked per upload.

packages/adapter-local/src/index.ts

@@ -49,11 +49,20 @@ export type LocalInitializeOptions = {
   timezone: string;
   /** Entity ID of the stack owner. */
   entityId: string;
+  /** Bypass the storage-ownership lock check. See LocalOpenOptions.force. */
+  force?: boolean;
 };
 
 export type LocalOpenOptions = {
   /** Absolute path to an existing .db file. */
   path: string;
+  /**
+   * Open even if a lock file from another live process is present.
+   * Only needed if that process is gone but its PID was reused by
+   * something else (the automatic stale-lock check already reclaims
+   * locks whose owning process is no longer running).
+   */
+  force?: boolean;
 };
 
 // -------------------------------------------------------
@@ -79,6 +88,7 @@ export class LocalAdapter implements StackAdapter {
       path: opts.path,
       entityId: opts.entityId,
       timezone: opts.timezone,
+      force: opts.force,
     });
     const blob = new DiskBlobAdapter(join(dirname(opts.path), 'attachments'));
     return new LocalAdapter(record, blob);
@@ -89,7 +99,7 @@ export class LocalAdapter implements StackAdapter {
    * use initialize() for new stacks.
    */
   static async open(opts: LocalOpenOptions): Promise<LocalAdapter> {
-    const record = await SQLiteRecordAdapter.open({ path: opts.path });
+    const record = await SQLiteRecordAdapter.open({ path: opts.path, force: opts.force });
     const blob = new DiskBlobAdapter(join(dirname(opts.path), 'attachments'));
     return new LocalAdapter(record, blob);
   }

packages/record-adapter-sqljs/src/index.ts

@@ -6,9 +6,15 @@
  * without native compilation.
  *
  * The database is held in memory and flushed to disk on every
- * write. Stack config (ownerEntityId, timezone) is stored as
- * a singleton _config@1 record with id='_config' in the
- * records table.
+ * write, atomically (temp file + rename). Stack config
+ * (ownerEntityId, timezone) is stored as a singleton _config@1
+ * record with id='_config' in the records table.
+ *
+ * A stack file is owned by exactly one process at a time (see
+ * docs/spec.md § Concurrency & storage ownership). open()/
+ * initialize() acquire a PID-stamped lock file beside the
+ * database and reject if another live process already holds
+ * it; close() releases it.
  *
  * Also exposes token management methods (createToken,
  * lookupToken, listTokens, revokeToken) used by server
@@ -18,7 +24,8 @@
 import initSqlJs from 'sql.js';
 import type { Database, SqlJsStatic } from 'sql.js';
 import { createHash, randomBytes } from 'node:crypto';
-import { readFileSync, writeFileSync, existsSync } from 'fs';
+import { readFileSync, writeFileSync, existsSync, renameSync, unlinkSync } from 'fs';
+import { dirname, basename, join } from 'path';
 import type {
   StackRecordAdapter,
   StackRecord,
@@ -44,11 +51,20 @@ export type SQLiteRecordInitializeOptions = {
   timezone: string;
   /** Entity ID of the stack owner. */
   entityId: string;
+  /** Bypass the storage-ownership lock check. See SQLiteRecordOpenOptions.force. */
+  force?: boolean;
 };
 
 export type SQLiteRecordOpenOptions = {
   /** Absolute path to an existing .db file. */
   path: string;
+  /**
+   * Open even if a lock file from another live process is present.
+   * Only needed if that process is gone but its PID was reused by
+   * something else (the automatic stale-lock check already reclaims
+   * locks whose owning process is no longer running).
+   */
+  force?: boolean;
 };
 
 export type TokenInfo = {
@@ -211,6 +227,56 @@ const rowToVersion = (row: Record<string, unknown>): RecordVersion => {
   return v;
 };
 
+// -------------------------------------------------------
+// Storage ownership lock
+// -------------------------------------------------------
+//
+// A stack file is owned by exactly one process at a time (see
+// docs/spec.md § Concurrency & storage ownership). The lock file
+// sits beside the database and records the PID of its opener.
+
+type LockInfo = { pid: number };
+
+const lockPathFor = (dbPath: string): string => `${dbPath}.lock`;
+
+const isProcessAlive = (pid: number): boolean => {
+  try {
+    process.kill(pid, 0);
+    return true;
+  } catch (err) {
+    // ESRCH: no such process — safe to reclaim. Anything else (e.g. EPERM,
+    // meaning the process exists but we can't signal it) — assume alive.
+    return (err as NodeJS.ErrnoException).code !== 'ESRCH';
+  }
+};
+
+const acquireLock = (dbPath: string, force?: boolean): void => {
+  const lockPath = lockPathFor(dbPath);
+  if (existsSync(lockPath)) {
+    const info = JSON.parse(readFileSync(lockPath, 'utf-8')) as LockInfo;
+    const ownedBySelf = info.pid === process.pid;
+    if (!ownedBySelf && !force && isProcessAlive(info.pid)) {
+      throw new Error(
+        `Stack database at "${dbPath}" is in use by another process (pid ${info.pid}). ` +
+          `Connect via its server instead, or pass { force: true } to override.`,
+      );
+    }
+  }
+  writeFileSync(lockPath, JSON.stringify({ pid: process.pid } satisfies LockInfo));
+};
+
+const releaseLock = (dbPath: string): void => {
+  const lockPath = lockPathFor(dbPath);
+  if (!existsSync(lockPath)) return;
+  try {
+    const info = JSON.parse(readFileSync(lockPath, 'utf-8')) as LockInfo;
+    if (info.pid === process.pid) unlinkSync(lockPath);
+  } catch {
+    // Corrupt lock file — remove it rather than leaving storage permanently unopenable.
+    unlinkSync(lockPath);
+  }
+};
+
 // -------------------------------------------------------
 // FTS4 query sanitization
 // -------------------------------------------------------
@@ -471,6 +537,7 @@ export class SQLiteRecordAdapter implements StackRecordAdapter {
           `Use SQLiteRecordAdapter.open() instead.`,
       );
     }
+    acquireLock(opts.path, opts.force);
     const SQL = await initSqlJs();
     const adapter = new SQLiteRecordAdapter(SQL, opts.path);
     adapter.db = new SQL.Database();
@@ -498,6 +565,7 @@ export class SQLiteRecordAdapter implements StackRecordAdapter {
           `Use SQLiteRecordAdapter.initialize() to create one.`,
       );
     }
+    acquireLock(opts.path, opts.force);
     const SQL = await initSqlJs();
     const adapter = new SQLiteRecordAdapter(SQL, opts.path);
     const fileBuffer = readFileSync(opts.path);
@@ -511,10 +579,20 @@ export class SQLiteRecordAdapter implements StackRecordAdapter {
   // Persistence
   // -------------------------------------------------------
 
-  /** Flush the in-memory database to disk. Called after every write. */
+  /**
+   * Flush the in-memory database to disk. Called after every write.
+   * Writes to a temp file in the same directory and renames it over
+   * the target — rename is atomic on POSIX, so a crash mid-write can
+   * never leave a torn, unreadable database file.
+   */
   private persist(): void {
     const data = this.db.export();
-    writeFileSync(this.path, Buffer.from(data));
+    const tmpPath = join(
+      dirname(this.path),
+      `.${basename(this.path)}.tmp-${randomBytes(6).toString('hex')}`,
+    );
+    writeFileSync(tmpPath, Buffer.from(data));
+    renameSync(tmpPath, this.path);
   }
 
   private readConfig(): void {
@@ -927,5 +1005,6 @@ export class SQLiteRecordAdapter implements StackRecordAdapter {
 
   async close(): Promise<void> {
     this.db.close();
+    releaseLock(this.path);
   }
 }

packages/record-adapter-sqljs/tests/record.test.ts

@@ -1,5 +1,5 @@
-import { describe, test, expect, beforeEach, afterEach } from 'vitest';
-import { mkdirSync, rmSync, existsSync } from 'fs';
+import { describe, test, expect, beforeEach, afterEach, vi } from 'vitest';
+import { mkdirSync, rmSync, existsSync, readdirSync, writeFileSync } from 'fs';
 import { join } from 'path';
 import { tmpdir } from 'os';
 import { SQLiteRecordAdapter } from '../src/index.js';
@@ -108,6 +108,94 @@ test('preserves ownerEntityId and timezone across reopen', async () => {
   expect(adapter.timezone).toBe('Europe/London');
 });
 
+// -------------------------------------------------------
+// Storage ownership lock
+// -------------------------------------------------------
+
+describe('storage ownership lock', () => {
+  afterEach(() => {
+    vi.restoreAllMocks();
+  });
+
+  test('open() rejects when a live process already holds the lock', async () => {
+    await initAdapter();
+    const otherPid = process.pid + 1;
+    writeFileSync(`${dbPath}.lock`, JSON.stringify({ pid: otherPid }));
+    vi.spyOn(process, 'kill').mockImplementation(() => true);
+
+    await expect(SQLiteRecordAdapter.open({ path: dbPath })).rejects.toThrow(
+      new RegExp(`in use by another process \\(pid ${otherPid}\\)`),
+    );
+  });
+
+  test('open() reclaims a lock left by a dead process', async () => {
+    await initAdapter();
+    const deadPid = process.pid + 1;
+    writeFileSync(`${dbPath}.lock`, JSON.stringify({ pid: deadPid }));
+    vi.spyOn(process, 'kill').mockImplementation(() => {
+      const err = new Error('no such process') as NodeJS.ErrnoException;
+      err.code = 'ESRCH';
+      throw err;
+    });
+
+    const adapter = await SQLiteRecordAdapter.open({ path: dbPath });
+    expect(adapter.ownerEntityId).toBe('entity-123');
+  });
+
+  test('open() with force bypasses a live lock', async () => {
+    await initAdapter();
+    const otherPid = process.pid + 1;
+    writeFileSync(`${dbPath}.lock`, JSON.stringify({ pid: otherPid }));
+    vi.spyOn(process, 'kill').mockImplementation(() => true);
+
+    const adapter = await SQLiteRecordAdapter.open({ path: dbPath, force: true });
+    expect(adapter.ownerEntityId).toBe('entity-123');
+  });
+
+  test('close() releases the lock so a later open() succeeds without force', async () => {
+    const adapter = await initAdapter();
+    await adapter.close();
+    await expect(SQLiteRecordAdapter.open({ path: dbPath })).resolves.toBeDefined();
+  });
+
+  test('reopening from the same process does not require force', async () => {
+    await initAdapter();
+    await expect(SQLiteRecordAdapter.open({ path: dbPath })).resolves.toBeDefined();
+  });
+
+  test('initialize() rejects when a live process holds the lock at the target path', async () => {
+    const otherPid = process.pid + 1;
+    writeFileSync(`${dbPath}.lock`, JSON.stringify({ pid: otherPid }));
+    vi.spyOn(process, 'kill').mockImplementation(() => true);
+
+    await expect(initAdapter()).rejects.toThrow(/in use by another process/);
+  });
+});
+
+// -------------------------------------------------------
+// Atomic persist
+// -------------------------------------------------------
+
+describe('atomic persist', () => {
+  test('leaves no temp files behind after a write', async () => {
+    const adapter = await initAdapter();
+    await adapter.createRecord(makeRecord());
+    const entries = readdirSync(testDir);
+    expect(entries.some((f) => f.includes('.tmp-'))).toBe(false);
+  });
+
+  test('the database file is valid after multiple persists', async () => {
+    const adapter = await initAdapter();
+    await adapter.createRecord(makeRecord({ id: 'r1' }));
+    await adapter.createRecord(makeRecord({ id: 'r2' }));
+    await adapter.patchContent('r1', { text: 'updated' });
+
+    const reopened = await SQLiteRecordAdapter.open({ path: dbPath });
+    expect((await reopened.getRecord('r1'))?.content).toEqual({ text: 'updated' });
+    expect(await reopened.getRecord('r2')).not.toBeNull();
+  });
+});
+
 // -------------------------------------------------------
 // Types
 // -------------------------------------------------------