Deferred design note from the server design review (haverstack/server#46 is the bounded-buffer symptom fix; this is the contract-level cause). Filed to be on record with explicit wake-up triggers — not proposed for the current implementation phase.
Problem
The blob contract speaks whole byte arrays:
putAttachment(data: Uint8Array): Promise<FileId>;
getAttachment(fileId: FileId): Promise<Uint8Array>;
The wire is not the problem — POST /attachments takes a raw binary body and downloads return one; both are streaming-shaped at the HTTP level. It's this contract in the middle that forces every layer above it to materialize the full payload: the server buffers uploads to call putAttachment and buffers downloads because it receives a complete Uint8Array (whose byteLength is also where Content-Length and the metadata record's size come from). Same shape propagates through Stack/ScopedStack.putAttachment and adapter-api.
Consequences at scale: a served stack's memory floor is roughly concurrent transfers × payload size × ~2. Tolerable at the 50 MB default ceiling; not tolerable the moment the ceiling rises for real media — and self-hosting on small boxes is this project's deployment story. Whole-buffer download also forecloses HTTP Range (Accept-Ranges/206), which is what seekable video/audio playback needs.
Sketch (additive, capability-flagged)
putAttachmentStream(data: ReadableStream<Uint8Array>): Promise<{ fileId: FileId; size: number }>;
getAttachmentStream(fileId: FileId, opts?: { offset?: number; length?: number }): Promise<{ stream: ReadableStream<Uint8Array>; size: number }>;Why deferred, and what wakes it
Unlike the wire-contract issues, waiting here bakes in no breakage — the change stays additive whenever wanted, and the wire is already the right shape. Wake-up triggers, any one sufficient:
- Raising the attachment ceiling past ~100 MB (or a real media use case: video playback wanting Range)
- Memory-constrained self-hosting reports
- A browser/OPFS blob adapter getting scheduled (forces the incremental-hash decision anyway)
One cheap thing worth doing now: haverstack/server#45's upload-gating fix will likely add a scoped bytes-entry point to core — give that new signature room to accept Uint8Array | ReadableStream<Uint8Array> later without another round of churn.
Work items (when woken)
Refs #45 (atomic-rename idiom), #46 (adapter packaging), #64 (capability-flagged adapter addition precedent), #56 (maxAttachmentBytes discovery — enforceable mid-stream once this lands), #57 (additive-in-place policy), haverstack/server#46 (symptom fix), haverstack/server#45 (signature coordination).
Deferred design note from the server design review (haverstack/server#46 is the bounded-buffer symptom fix; this is the contract-level cause). Filed to be on record with explicit wake-up triggers — not proposed for the current implementation phase.
Problem
The blob contract speaks whole byte arrays:
The wire is not the problem —
POST /attachmentstakes a raw binary body and downloads return one; both are streaming-shaped at the HTTP level. It's this contract in the middle that forces every layer above it to materialize the full payload: the server buffers uploads to callputAttachmentand buffers downloads because it receives a completeUint8Array(whosebyteLengthis also whereContent-Lengthand the metadata record'ssizecome from). Same shape propagates throughStack/ScopedStack.putAttachmentandadapter-api.Consequences at scale: a served stack's memory floor is roughly
concurrent transfers × payload size × ~2. Tolerable at the 50 MB default ceiling; not tolerable the moment the ceiling rises for real media — and self-hosting on small boxes is this project's deployment story. Whole-buffer download also forecloses HTTP Range (Accept-Ranges/206), which is what seekable video/audio playback needs.Sketch (additive, capability-flagged)
fsconverts viaReadable.toWeb). Core ships buffering shims both directions so existing adapters keep working — additive in place per the standing policy (_grant records store the grantee in entityId, the authorship field — move it into GrantContent #57), flagged like Orphaned blob GC: deleting the last referencing record leaves bytes on disk forever #64'slistFiles().putAttachmentStreamreturnssizebecause the caller no longer holds the bytes and the_attachment@1record needs it.crypto.subtle.digestwants the whole buffer). Node'screateHashstreams fine, soblob-adapter-diskis unaffected — but a future browser/OPFS blob adapter needs a small JS/WASM SHA-256 or falls back to the buffering shim. Settle this when that adapter gets scheduled.listFiles(),ScopedStack's access predicate runs before bytes flow. Wire needs zero changes for streaming itself; Range support is additive HTTP (future PATCH contract mismatch: spec defines content-only merge patch, APIAdapter sends record-field envelopes #52 fixtures if/when specced).Why deferred, and what wakes it
Unlike the wire-contract issues, waiting here bakes in no breakage — the change stays additive whenever wanted, and the wire is already the right shape. Wake-up triggers, any one sufficient:
One cheap thing worth doing now: haverstack/server#45's upload-gating fix will likely add a scoped bytes-entry point to core — give that new signature room to accept
Uint8Array | ReadableStream<Uint8Array>later without another round of churn.Work items (when woken)
blob-adapter-disk: temp-file + incremental-hash + rename implementation;offset/lengthreadsContent-Lengthfrom metadata/stat; Range support as follow-upRefs #45 (atomic-rename idiom), #46 (adapter packaging), #64 (capability-flagged adapter addition precedent), #56 (
maxAttachmentBytesdiscovery — enforceable mid-stream once this lands), #57 (additive-in-place policy), haverstack/server#46 (symptom fix), haverstack/server#45 (signature coordination).