Adopt SDK parsed-document storage for chunk reads, sync, and chat assets#127
Merged
suguanYang merged 12 commits intoJul 5, 2026
Merged
Conversation
Replace the hand-rolled Blob snapshot machinery with the SDK's ParsedDocumentStorage: a Vercel-Blob adapter, a QStash-backed resumable sync scheduler + /api/sources/parsed-sync route, and chunk reads routed through knowledge.readChunks/getDocumentOutline (assetUrlPolicy:"durable"). Parse flow syncs to storage before markReady and stages sync failures as failure_stage="storage_sync"; retry resumes sync without reparsing. Chat citation hardening is unified onto the durable read path. Adds migration 0011 (revision_key/sync_status/sync_error on source_parse_results, failure_stage on sources). Deletes chunks/server, remote-parsed-snapshot-sync, and parse-result-storage-adapter. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates the Notebook onto the Knowhere SDK's
ParsedDocumentStoragemodel so chunk read/display/sync and chat citation-asset hardening share one path with the SDK/CLI/MCP, replacing the hand-rolled Blob snapshot machinery.ParsedDocumentStorageadapter (parsed-document-blob-storage.ts) keyedworkspaces/{ws}/parsed-documents/{documentId}/{revisionKey}/…with traversal guards and an injectable blob-store seam.parsed-document-sync-scheduler.ts+parsed-sync-route-workflow.ts+POST /api/sources/parsed-sync, reusing the deterministic-workflowRunIdcontinuation pattern; loopssyncParsedDocumentuntilcompleted.makeKnowhereClientWithParsedStorageconfigureswithParsedStorage({storage, scheduler, limits})with Vercel-safe limits.route-chunks.tsdelegates toknowledge.readChunks({assetUrlPolicy:"durable"})(paged + load-all) andgetDocumentOutline, viachunks/read.ts. Gating isstatus !== "ready" → 202, else always serve (remote fallback guarantees non-empty for ready docs).markReady; sync failures stage asfailure_stage: "storage_sync". Retry resumes the sync from the storeddocumentId+revisionKeywithout reparsing.media-asset-hardening.tsshrinks to owned-URL passthrough + durable-map resolution; agenticretrieval.queryunchanged).0011addsrevision_key/sync_status/sync_errortosource_parse_resultsandfailure_stagetosources; relaxesresult_blob_urlto nullable.chunks/server.ts,remote-parsed-snapshot-sync.ts,parse-result-storage-adapter.ts.Also includes the page-citation-asset support commits this branch builds on.
Testing
pnpm typecheck— cleanpnpm lint— cleanpnpm test— 618 passed, 14 skippedpnpm db:generate— confirms only0011added, no driftsyncParsedDocument→ storage-hit fast path, plus the Notebookread.tshelpers (page read, load-all, durable asset map).Blocked / not covered
pnpm dev: upload → ready → chunk panel → chat citation) not run here.🤖 Generated with Claude Code
Companion PRs (coordinated parsed-storage rollout):