Skip to content

JS: Add lazy streaming SELECT/CONSTRUCT query cursors#2

Merged
rybesh merged 1 commit into
mainfrom
streaming-query-cursors
Jun 11, 2026
Merged

JS: Add lazy streaming SELECT/CONSTRUCT query cursors#2
rybesh merged 1 commit into
mainfrom
streaming-query-cursors

Conversation

@rybesh

@rybesh rybesh commented Jun 11, 2026

Copy link
Copy Markdown
Member

Summary

Adds lazy streaming query cursors to the JS/WASM binding so large result sets can be pulled in bounded batches instead of being materialized into an Array or serialized into one string (which can exceed V8's ~512 MB max string length):

  • Store.querySolutions(query, options?) -> QuerySolutions (SELECT)
  • Store.queryTriples(query, options?) -> QueryTriples (CONSTRUCT/DESCRIBE)

Each cursor exposes nextBatch(count) (an empty array signals exhaustion; count must be >= 1) and, for SELECT, a variables getter. The wasm-bindgen-generated free() releases the cursor.

This is the productionized form of the spike in #1: SELECT and CONSTRUCT, full option parity, TS typings, docs, and tests.

How it works

The iterators are 'static because on_store() snapshots storage rather than borrowing &Store, so a cursor lives in a wasm-bindgen struct across JS calls and is MVCC-isolated — an open cursor keeps yielding the rows that matched at query time even if the store is concurrently mutated. The snapshot is pinned until free().

Option parsing is shared with query() via a new execute_query() helper, so the cursors get full parity (base_iri, default_graph, named_graphs, use_default_graph_as_union) with no behavior change to query().

Why

Downstream dkglab/graph-store#50 / british-music-trade/spaestiem#103: the worker serializes a SELECT into one string and hits the V8 cap on large datasets (silent empty table on a 200 MB fixture). The binding-level primitive both downstream consumer shapes need (random-access window + one-pass stream) is exactly this batched pull.

Testing

Red→green TDD. js/test/store.test.ts adds #querySolutions() (14) and #queryTriples() (8) tests covering bounded batching, stable exhaustion, full option parity vs query(), term/quad fidelity, non-matching-form rejection, MVCC snapshot isolation across a mid-iteration DELETE, and free(). Full suite: 65 passed (2 files); cargo clippy and biome clean.

🤖 Generated with Claude Code

Add Store.querySolutions(query, options) -> QuerySolutions (SELECT) and
Store.queryTriples(query, options) -> QueryTriples (CONSTRUCT/DESCRIBE).
Each cursor holds the lazy QuerySolutionIter/QueryTripleIter alive instead
of draining it into an Array, exposing nextBatch(count) (an empty Array
signals exhaustion; count must be >= 1) plus, for SELECT, a variables
getter. This lets the WASM bindings stream large result sets in bounded
batches rather than materializing the whole set or serializing it into one
string that can exceed V8's max string length.

The iterators are 'static because on_store() snapshots storage rather than
borrowing &Store, so a cursor lives in a wasm-bindgen struct across JS calls
and is MVCC-isolated: an open cursor still yields the rows that matched at
query time even if the store is concurrently mutated. The snapshot is pinned
until the cursor's generated free() is called.

Option parsing is shared with query() via a new execute_query() helper, so
the cursors get full parity (base_iri, default_graph, named_graphs,
use_default_graph_as_union) with no behavior change to query(). Adds
TypeScript typings for both methods and cursor classes, README docs, and
red/green tests covering bounded batching, exhaustion, option parity, term
fidelity, non-matching-form rejection, snapshot isolation, and free().

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@rybesh rybesh closed this Jun 11, 2026
@rybesh rybesh reopened this Jun 11, 2026
@rybesh rybesh merged commit 88b0f7d into main Jun 11, 2026
38 of 42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant