Skip to content

Add parsed-document storage model to SDK and MCP#111

Merged
suguanYang merged 10 commits into
mainfrom
feature/wangbinqi/page-citation-assets
Jul 5, 2026
Merged

Add parsed-document storage model to SDK and MCP#111
suguanYang merged 10 commits into
mainfrom
feature/wangbinqi/page-citation-assets

Conversation

@suguanYang

@suguanYang suguanYang commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Introduces the shared parsed-document storage model consumed by the CLI, MCP, and Notebook, plus page-citation asset support.

SDK (src/)

  • ParsedDocumentStorage interface + DiskParsedDocumentStorage reference impl (src/storage/): manifest / chunk pages / assets / resumable sync-progress, keyed by documentId + revisionKey (revisionKey = jobResultId ?? jobId).
  • knowledge.withParsedStorage({storage, scheduler?, limits?}) — reads try configured storage first (fresh iff manifest.revisionKey === revisionKey), fall back to remote documents.listChunks, return the bounded result immediately, and schedule a background sync on a miss.
  • readChunks (paged display + assetUrlPolicy:"none"|"durable"), grepChunks (single-document, with continuation cursor), getDocumentOutline.
  • syncParsedDocument — resumable, bounded by limits, commits the manifest only after all pages are present.
  • Durable asset hardening: with assetUrlPolicy:"durable" + configured storage, visible chunk asset URLs (primary + metadata.pageAssets[]) are written into storage and rewritten; on failure the URL is omitted rather than exposing a presigned Knowhere URL.
  • importJobResult (replaces deprecated cacheJobResult), loadJobResult sync-to-storage, page-citation asset generation + Piscina-backed PDF page renderer.

MCP (packages/mcp/)

  • Updated to delegate to readChunks/grepChunks/getDocumentOutline with storage-first, remote-fallback behavior; descriptions and tests updated.

Testing

  • SDK unit tests cover storage round-trips, storage-hit vs. remote-fallback reads, non-blocking sync scheduling, paged readChunks + invalid param combinations, single-document bounded grep with truncation, and outline fallback.
  • Verified live against a local Knowhere API: documents.listChunks snake→camel transform, remote read → scheduler fire → syncParsedDocument → storage-hit fast path, getDocumentOutline, and grepChunks.

🤖 Generated with Claude Code


Companion PRs (coordinated parsed-storage rollout):

@suguanYang
suguanYang merged commit 0d2d472 into main Jul 5, 2026
4 checks passed
@github-actions github-actions Bot mentioned this pull request Jul 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant