JS: Add lazy streaming SELECT/CONSTRUCT query cursors#2
Merged
Conversation
Add Store.querySolutions(query, options) -> QuerySolutions (SELECT) and Store.queryTriples(query, options) -> QueryTriples (CONSTRUCT/DESCRIBE). Each cursor holds the lazy QuerySolutionIter/QueryTripleIter alive instead of draining it into an Array, exposing nextBatch(count) (an empty Array signals exhaustion; count must be >= 1) plus, for SELECT, a variables getter. This lets the WASM bindings stream large result sets in bounded batches rather than materializing the whole set or serializing it into one string that can exceed V8's max string length. The iterators are 'static because on_store() snapshots storage rather than borrowing &Store, so a cursor lives in a wasm-bindgen struct across JS calls and is MVCC-isolated: an open cursor still yields the rows that matched at query time even if the store is concurrently mutated. The snapshot is pinned until the cursor's generated free() is called. Option parsing is shared with query() via a new execute_query() helper, so the cursors get full parity (base_iri, default_graph, named_graphs, use_default_graph_as_union) with no behavior change to query(). Adds TypeScript typings for both methods and cursor classes, README docs, and red/green tests covering bounded batching, exhaustion, option parity, term fidelity, non-matching-form rejection, snapshot isolation, and free(). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds lazy streaming query cursors to the JS/WASM binding so large result sets can be pulled in bounded batches instead of being materialized into an
Arrayor serialized into one string (which can exceed V8's ~512 MB max string length):Store.querySolutions(query, options?) -> QuerySolutions(SELECT)Store.queryTriples(query, options?) -> QueryTriples(CONSTRUCT/DESCRIBE)Each cursor exposes
nextBatch(count)(an empty array signals exhaustion;countmust be>= 1) and, for SELECT, avariablesgetter. The wasm-bindgen-generatedfree()releases the cursor.This is the productionized form of the spike in #1: SELECT and CONSTRUCT, full option parity, TS typings, docs, and tests.
How it works
The iterators are
'staticbecauseon_store()snapshots storage rather than borrowing&Store, so a cursor lives in a wasm-bindgen struct across JS calls and is MVCC-isolated — an open cursor keeps yielding the rows that matched at query time even if the store is concurrently mutated. The snapshot is pinned untilfree().Option parsing is shared with
query()via a newexecute_query()helper, so the cursors get full parity (base_iri,default_graph,named_graphs,use_default_graph_as_union) with no behavior change toquery().Why
Downstream
dkglab/graph-store#50/british-music-trade/spaestiem#103: the worker serializes a SELECT into one string and hits the V8 cap on large datasets (silent empty table on a 200 MB fixture). The binding-level primitive both downstream consumer shapes need (random-access window + one-pass stream) is exactly this batched pull.Testing
Red→green TDD.
js/test/store.test.tsadds#querySolutions()(14) and#queryTriples()(8) tests covering bounded batching, stable exhaustion, full option parity vsquery(), term/quad fidelity, non-matching-form rejection, MVCC snapshot isolation across a mid-iterationDELETE, andfree(). Full suite:65 passed (2 files);cargo clippyand biome clean.🤖 Generated with Claude Code