feat: distributed cursor loadMore() + real MCP pagination + docs sweep — Query Phase 4 (SPEC-306a/b/c)#48
Merged
Merged
Conversation
…QueryResult
- cursor.rs: add cursor_query_hashes (single authoritative hash source called by
both consume-side and emit-side) and build_next_cursor (hash-source-agnostic
single emission helper)
- converter.rs: refactor build_cursor_vertex_config to call cursor_query_hashes
(removes private hash_debug closure); write limit+1 into DAG limit-vertex config
so the limit+1 sentinel flows to the emission site
- coordinator.rs: introduce #[derive(Debug)] DistributedQueryResult { rows, has_more };
apply_global_sort_and_limit fetches limit+1 globally, absorbs sentinel, truncates
to limit, returns struct; execute_distributed and combine_group_by_results now
return DistributedQueryResult; migrate all 5 coordinator-internal test sites to
.rows/.has_more; add absorb_sentinel helper for single-node bypass
- http_sync.rs: refactor inline cursor-encode to call build_next_cursor (single
emission path, zero duplication)
- query.rs: wire coordinator branch to DistributedQueryResult; capture
cursor_query_hashes + query_limit + sort_values_template before query move;
add coordinator_has_more + local_has_more tracking; local/predicate branch
detects has_more via limit+1 sentinel and truncates; reconcile has_more before
max_query_records clamp; emit real next_cursor via build_next_cursor; set
cursor_status from input cursor presence
…m tests - query.rs::handle_query_subscribe: capture cursor_query_hashes before query is moved; coordinator branch reads dist_result.rows/has_more; local DAG branch detects has_more from limit+1 sentinel and truncates; call build_next_cursor as single emission site; emit cursor_status - sim/cluster.rs: migrate 2 existing distributed tests to .rows/.has_more; add paged_ws_query_under_partition_emits_cursor_and_correct_follow_up (2-node, proper partition-aware data distribution, fault injection); add single_node_cursor_roundtrip_has_more_flips_at_exhaustion (AC4 hash-match) - cargo fmt: apply formatting to converter.rs, coordinator.rs, http_sync.rs All 10 sim tests pass; 1293 lib tests pass; clippy clean; fmt clean. AC7: no next_cursor: None remains in query.rs.
- Add classify_cursor_status helper in query/cursor.rs that reuses the same expiry + hash checks the DAG CursorProcessor applies, against the query's cursor_query_hashes — so the emitted status agrees with the DAG accept/reject decision by construction (deterministic, not inferred from an empty result page) - query.rs::handle_query_subscribe decodes the input cursor before the move and reports None/Expired/Invalid/Valid via the helper instead of always Valid - Add 4 unit tests for classify_cursor_status (none/valid/invalid/expiry-precedence) Closes the AC2 gap: clients can now distinguish a stale pagination token (restart) from genuine exhaustion.
- Add private mergePageResults() that only adds/updates keys, never removes existing keys — preserves results from prior pages absent in the new batch - Add public loadMore() that reads nextCursor/hasMore from _paginationInfo, creates a temporary one-shot QueryHandle with the cursor, waits for settlement, then merges results via mergePageResults() and advances pagination state - Add _loadMoreInFlight string latch to deduplicate concurrent calls for the same cursor token
- Add QueryOncePagedResult<T> interface exported from TopGunClient and index.ts - Add queryOncePaged() with typed overloads mirroring queryOnce pattern; after settlement reads handle.getPaginationInfo() and returns paged result - Add handleUnsettledPaged() private helper for offline/timeout error handling - queryOnce return type and behavior are left entirely unchanged
…oncurrency guard
- QueryHandle.test.ts: add loadMore() describe block with 4 tests:
- appends page-2 rows without pruning page-1 rows (disjoint keys)
- is a no-op when hasMore is false (no duplicate subscribeToQuery)
- concurrent calls are deduplicated via in-flight latch (only 1 temp handle)
- advances paginationInfo to the new page cursor after successful load
- QueryOnce.test.ts: add queryOncePaged describe block with 7 tests:
- resolves to { items, cursor, hasMore } from server response
- hasMore false and cursor undefined when no further pages
- auto-unsubscribes after resolving
- default offline throws QueryOnceUnsettledError
- default timeout throws QueryOnceUnsettledError
- allowLocal offline throws QueryOnceLocalError carrying local snapshot
- Add loadMore(): Promise<void> to UseQueryResult<T> interface - Implement useCallback-memoized loadMore delegating to handleRef.current - Guard: no-op when hasMore is false or handle is null (returns Promise.resolve()) - Add loadMore to useMemo return with paginationInfo.hasMore in dep array
- Add mock loadMore jest.fn() to the shared QueryHandle mock - Test: loadMore() delegates to handle.loadMore() when hasMore is true - Test: loadMore() is a no-op when hasMore is false - Test: loadMore ref is stable across re-renders when hasMore stays true - Fix: apply prettier formatting to mcp-server query.ts (pre-existing format drift)
…sor arg) - Replace queryOnce +1 probe trick with queryOncePaged returning server-authoritative hasMore + cursor - Add optional cursor field to QueryArgsSchema and toolSchemas.query JSON schema - Thread cursor through queryFilter.cursor when provided by agent - Surface real continuation cursor token in tool result when hasMore is true - Replace "no cursor to page through" note with actionable next-page cursor instruction - Keep narrow-query guidance alongside the cursor info
…ransition Locks in correct useCallback semantics: callback identity must change when hasMore flips so the new closure observes hasMore:true and delegates rather than staying a stale no-op.
Rewrites the stale cursor-pagination example in client.mdx to the shipped
loadMore() / paged-query API (SPEC-306b), adds loadMore() to QueryHandle<T>
method list, documents client.sql() as opt-in/server-feature-gated with the
joins/window/HAVING/DISTINCT niche, and corrects mcp.mdx topgun_query to
reflect the real cursor parameter + hasMore continuation note.
Doc-vs-behavior sweep checklist (TODO-438):
[x] client.mdx loadMore() — ships in QueryHandle.ts:446
[x] client.mdx getPaginationInfo() — ships in QueryHandle.ts
[x] client.mdx cursor example uses sort (required for keyset cursor build)
[x] client.mdx sql() opt-in/OFF-by-default/niche — ships in TopGunClient.ts:1070 + SqlClient.ts
[x] client.mdx sql() niche (joins/window/HAVING/DISTINCT, group-by on DAG) — matches #[cfg(feature="datafusion")]
[x] mcp.mdx topgun_query cursor arg — matches query.ts:34 (QueryArgsSchema)
[x] mcp.mdx hasMore continuation note — matches query.ts:139-147 (continuationNote)
[x] mcp.mdx cursor in parameters list — matches QueryArgsSchema
[x] Cargo.toml default=["redb"] unchanged — confirmed at line 12
[x] datafusion NOT in default features — confirmed (optional only)
[x] README — no update needed (no query feature matrix exists)
AC2 execution (cursor pagination against local server with 306a binary):
Server: topgun-server release (built from 306a commits)
12 records in 'items' map, limit=4, sort={score:'asc'}
Page 1: items=[item-001..004], hasMore=true, cursor=YES
Page 2: items=[item-005..008], hasMore=true, cursor=YES (via p1.cursor)
Page 3: items=[item-009..012], hasMore=false (last page)
Overlap between pages: 0
loadMore() test (10 records, limit=3):
After settle: 3 items, hasMore=true
After loadMore(): 6 items (append-only merge, no duplicates)
After 2nd loadMore(): 9 items — multiple calls work
Demo server (demo.topgun.build): WebSocket non-101 from Node.js environment
(browser connections work; verified via /health endpoint returning ok)
Cargo.toml not touched (verify-only — constraint confirmed correct).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Query Engine Consolidation Phase 4 (from TODO-443, executed as SPEC-306a/b/c) — closes the
milestone. Exposes the distributed cursor end-to-end with an ergonomic
loadMore()surface,makes MCP pagination real, and brings docs in line with behavior.
Changes
306a — server (Rust): accurate distributed cursor —
query.rsemitsnext_cursor+cursor_status(valid/expired/invalid/none), limit+1 sentinel,DistributedQueryResult; cursorhelpers in
query/cursor.rs; coordinator/converter wiring; distributed sim tests.306b — client / react / mcp-server (TS):
QueryHandle.loadMore()(append-only merge) +queryOncePaged()returning{ items, cursor, hasMore }; memoizeduseQueryloadMore; MCPtopgun_queryreal cursor (removes the +1 probe, adds acursorarg) + behavioral/concurrencytests.
306c — docs: doc-vs-behavior sweep (TODO-438 exit gate) —
client.mdxleads withloadMore(),opt-in DataFusion SQL section,
mcp.mdxdocuments the realcursor/hasMore;llms-full.txtregenerated. Docs-only, zero Rust logic (
default = ["redb"]unchanged).Verification
Each sub-spec reviewed via the SpecFlow impl-review gate (306c clean APPROVED 0/0/0). Spans Rust +
TS + docs → CI gates: Rust (cargo test + sim + clippy
-D warnings+ fmt) ANDBuild · Test · Lint (client/react/mcp-server) AND docs-astro build.