SODA: streaming getCursor() for unbounded result sets (#213)#215
Merged
Conversation
getDocuments() materialises matches into host arrays, so it needs a capacity and raises past a cap. Add SodaOperation.getCursor(batchSize=100), which streams the matched documents via offset pagination — repeated skip/limit windows over the same find() — so an arbitrarily large result set fetches without choosing a cap. Honours an explicit skip / limit and stops on the first short batch. Returns an iterator (sync) / async iterator (async): for doc in coll.find().filter(qbe).getCursor(batchSize=500): ... async for doc in coll.find().getCursor(): ... getDocuments() and getCursor() now share one batch-fetch helper. Verified documents stream exactly once with no duplicate or skip across batches, and that filter / limit are honoured. Sync + async; 23ai/21c pass, 11g/9i skip. Closes #213 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Peter Lemenkov <lemenkov@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SODA follow-up: stream large result sets without the
getDocumentsmaterialisation cap.API (sync + async)
SodaOperation.getCursor(batchSize=100)→ an iterator ofSodaDocument, fetched in batches via offset pagination (repeatedskip/limitwindows over the samefind()). Honours an explicitskip/limit, stops on the first short batch.getDocuments()andgetCursor()now share one batch-fetch helper.Tests: documents stream exactly once with no duplicate or skip across batches; filter and limit are honoured; empty result yields nothing. Sync + async; 23ai/21c pass, 11g/9i skip. Offline 527.
Closes #213
🤖 Generated with Claude Code