Add parsed-document storage model to SDK and MCP

.changeset/page-citation-assets-metadata.md

@@ -0,0 +1,6 @@
+---
+"@ontos-ai/knowhere-sdk": major
+"@ontos-ai/knowhere-mcp": major
+---
+
+Preserve page citation asset descriptors only in chunk metadata, remove the deprecated SDK-side page citation asset generation options, and split local-cache/server-safe result handling into explicit `knowledge.parseToLocalCache(...)`, `knowledge.importJobResult(...)`, and `knowledge.loadJobResult(...)` methods.

.changeset/parsed-storage-redesign.md

@@ -0,0 +1,11 @@
+---
+"@ontos-ai/knowhere-sdk": minor
+"@ontos-ai/knowhere-mcp": minor
+---
+
+Move parsed document reads to a storage-first SDK model with remote chunk fallback.
+
+Adds parsed snapshot storage types, `knowledge.withParsedStorage(...)`,
+`knowledge.syncParsedDocument(...)`, paged `readChunks(...)` display params,
+durable asset URL hardening, bounded remote grep/outline fallback, and explicit
+disk parsed storage for MCP cache-directory usage.

.gitattributes

@@ -0,0 +1 @@
+*.pdf binary

.gitignore

@@ -1,6 +1,7 @@
 # Dependencies
 node_modules/
 yarn.lock
+.repos/effect
 
 # Build output
 dist/

README.md

@@ -23,7 +23,7 @@ npm install @ontos-ai/knowhere-sdk
 
 **Requirements:**
 
-- Node.js >= 20.19.0
+- Node.js >= 22.13.0
 - npm >= 10.0.0
 - TypeScript >= 5.0 (optional, for type checking)
 
@@ -145,6 +145,53 @@ const result = await client.parse({
 });
 ```
 
+### Page Citation Assets
+
+Page citation assets are generated by Knowhere during page-memory parsing. The
+SDK does not render PDF pages locally; it preserves the page image descriptors
+returned by the server on page chunks and retrieval metadata.
+
+```typescript
+import Knowhere from '@ontos-ai/knowhere-sdk';
+
+const client = new Knowhere({ apiKey: process.env.KNOWHERE_API_KEY });
+
+const result = await client.parse({
+  file: './manual.pdf',
+});
+
+for (const chunk of result.pageChunks) {
+  console.log(chunk.metadata.pageAssets);
+}
+```
+
+`metadata.pageAssets` entries point at concrete page images stored with the Knowhere
+result. `artifactRef` is the durable result artifact path; `assetUrl` is an
+optional server-generated access URL.
+
+```typescript
+type PageCitationAsset = {
+  pageNum: number;
+  artifactRef: string;
+  assetUrl?: string;
+  contentType: 'image/png' | 'image/jpeg';
+  width?: number;
+  height?: number;
+  source: 'knowhere-rendered-page-citation-source';
+};
+```
+
+Document chunk APIs can include signed page image URLs when requested:
+
+```typescript
+const chunks = await client.documents.listChunks('doc_123', {
+  chunkType: 'page',
+  includeAssetUrls: true,
+});
+
+console.log(chunks.chunks[0]?.metadata.pageAssets?.[0]?.assetUrl);
+```
+
 ### Low-Level API
 
 For granular control over the job lifecycle:
@@ -299,12 +346,16 @@ console.log(archived.status);
 
 ### Local Knowledge Tools
 
-The SDK can also keep parsed results in a local cache and run exact inspection
-tools over that cached copy. This is the implementation used by the separate
-`@ontos-ai/knowhere-mcp` package.
+The SDK can also run exact inspection tools over parsed results. Local import
+helpers still write under the SDK cache directory by design, while published
+`documentId` reads use configured parsed storage first and fall back to
+Knowhere's remote chunk API without importing the result ZIP into local disk.
+Server workflows can configure parsed storage with
+`client.knowledge.withParsedStorage(...)` so Notebook, MCP, and CLI surfaces
+share the same snapshot model.
 
 ```typescript
-const parsed = await client.knowledge.parse({
+const parsed = await client.knowledge.parseToLocalCache({
   file: './manual.pdf',
   localDocumentId: 'manual',
 });
@@ -334,27 +385,49 @@ console.log(grep.matches);
 console.log(serverSearch.references);
 ```
 
-Local grep and reads use the cached parse result, not server-side chunk scans.
-Search uses the Knowhere API retrieval query; local document IDs only help map
-returned server document IDs back to local cache IDs when available.
-If a search result only has a published `documentId`, or an async parse flow only
-has a completed `jobId`, read-oriented helpers can accept that remote identifier
-directly and will sync the result into the local cache before reading. For a
-`documentId`, the SDK resolves the document's current published `jobId` and then
-downloads the parser result ZIP through the same `cacheJobResult(...)` path:
+Local grep and reads use the cached parse result. Published `documentId` grep
+streams `documents.listChunks(...)` page by page when parsed storage is missing
+or stale. Search uses the Knowhere API retrieval query; local document IDs only
+help map returned server document IDs back to local cache IDs when available.
+When `knowledge.parseToLocalCache(...)`, `knowledge.importJobResult(...)`, or
+`knowledge.loadJobResult(...)` runs with configured parsed storage, the SDK
+writes chunk pages, durable assets, sync progress, and a committed
+`manifest/current.json` before returning. Partial snapshots are ignored until
+the manifest is committed.
 
 ```typescript
-const remoteRead = await client.knowledge.readChunks({
+const knowledge = client.knowledge.withParsedStorage({
+  storage: myParsedDocumentStorage,
+  scheduler: myBackgroundScheduler,
+  limits: { remotePageSize: 100, maxPagesPerSync: 10 },
+});
+```
+
+If a search result only has a published `documentId`, read-oriented helpers can
+accept that remote identifier directly. `readChunks` supports display paging;
+`assetUrlPolicy: 'durable'` hardens only returned assets through configured
+storage and omits asset URLs if hardening fails:
+
+```typescript
+const remoteRead = await knowledge.readChunks({
   documentId: 'doc_123',
-  sectionPath: 'Overview',
-  limit: 5,
+  page: 1,
+  pageSize: 20,
+  chunkType: 'page',
+  assetUrlPolicy: 'durable',
 });
 
-const remoteOutline = await client.knowledge.getDocumentOutline({
+const remoteOutline = await knowledge.getDocumentOutline({
   documentId: 'doc_123',
 });
 
-const jobRead = await client.knowledge.readChunks({
+const grep = await knowledge.grepChunks({
+  documentId: 'doc_123',
+  pattern: 'warranty',
+  maxResults: 10,
+});
+
+const jobRead = await knowledge.readChunks({
   jobId: jobResult.jobId,
   localDocumentId: 'manual',
   limit: 5,
@@ -383,9 +456,13 @@ if (status.job.isDone && status.cache.document) {
 
 When the job was started through `client.knowledge.startParse(...)`,
 `getJobStatus(...)` automatically caches the completed result locally the first
-time it observes `status.job.isDone`. Use `cacheJobResult(...)` only to recover a
-completed job that was not started through the local knowledge helper, or to
-retry a cache step explicitly.
+time it observes `status.job.isDone`. Use `importJobResult(...)` to recover a
+completed job into the local cache when it was not started through the local
+knowledge helper, or to retry a local import step explicitly. Use
+`loadJobResult(...)` for server workflows that should load a completed result
+without creating SDK local-disk cache state. Use `syncParsedDocument(...)` to
+explicitly resume or retry parsed-storage sync for an existing `documentId`,
+`jobId`, or local parsed result.
 
 Follow-up queries can exclude documents or sections for one request:
 

docs/new-design.md

@@ -0,0 +1,118 @@
+  # SDK Parsed Storage Redesign
+
+  ## Summary
+  Move parsed chunk read/sync behavior into the SDK so Notebook, CLI, and MCP share one model. Callers provide a parsed-document storage adapter; SDK reads storage first, falls back to Knowhere remote storage
+  transparently, and schedules non-blocking storage sync on read misses. Parse flow is different: after parsing succeeds, SDK must sync to provided storage before the caller treats the document as complete/
+  ready.
+
+  ## Public SDK Shape
+  - Keep `knowledge.readChunks`, `knowledge.grepChunks`, and `knowledge.getDocumentOutline` as the only public read methods; do not add public `readChunkPage`.
+  - Extend `readChunks` with paged display params:
+    - `page`, `pageSize`, `chunkType`, `assetUrlPolicy?: "none" | "durable"`.
+    - Paged mode must not combine with scan-heavy filters like `sectionPath`, `startChunk`, `endChunk`, or `chunkId`.
+  - Add storage configuration on the SDK/knowledge client, e.g. `knowledge.withParsedStorage({ storage, scheduler, limits })`.
+  - Replace new usage of `cacheJobResult` with clearer methods:
+    - `importJobResult` means explicit local import.
+    - `syncParsedDocument` means write a parsed snapshot/assets to the configured storage.
+    - Keep `cacheJobResult` only as deprecated compatibility alias.
+
+  ## SDK Implementation
+  - Define a `ParsedDocumentStorage` adapter for parsed snapshots, not just assets:
+    - read/write manifest
+    - read/write chunk pages
+    - read/write durable assets
+    - get asset URL
+    - read/write sync progress metadata
+  - Use `revisionKey = jobResultId ?? jobId` for freshness. Storage is valid only when its manifest revision matches the current remote revision.
+  - Read flow:
+    - resolve one document reference: `documentId`, `localDocumentId`, or `jobId`
+    - try configured storage first
+    - if missing/stale, read remote via `documents.listChunks`
+    - return the requested bounded result immediately
+    - schedule background sync into configured storage
+  - Parse flow:
+    - parse completes on Knowhere
+    - SDK writes chunks/assets/manifest to configured storage
+    - only then returns completion to the caller
+  - Background sync must be resumable and Vercel-safe:
+    - cursor/progress includes `documentId`, `revisionKey`, next chunk page, next asset index
+    - bounded by page count, asset count, and deadline per step
+    - final manifest is written only after all required pages/assets are present
+    - partial snapshots are ignored by reads until manifest commit
+
+  ## Grep And Outline
+  - `grepChunks` supports only one specific document, never broad workspace/search scope.
+  - Broad search continues to use Knowhere `retrieval.query`.
+  - Remote grep streams `documents.listChunks(..., includeAssetUrls: false)` page by page, applies literal/regex matching SDK-side, and stops at `maxResults`, deadline, max pages, or end of document.
+  - Grep response should include `truncated` and a continuation cursor when it stops early.
+  - `getDocumentOutline` reads full outline from storage when available. On remote fallback, build a lightweight section outline from chunk `sectionPath` within configured page/deadline limits and mark it
+  truncated if incomplete.
+
+  ## Notebook Changes
+  - Notebook uses SDK directly for chunk read/display/chat tooling; do not route this through MCP.
+  - Notebook provides a Vercel Blob `ParsedDocumentStorage` adapter and an Upstash/Vercel-safe scheduler.
+  - New Notebook parse/upload flow remains hidden behind `parsing`:
+    - Knowhere parse success plus Blob sync success -> mark source `ready`
+    - Knowhere parse failure -> mark source `failed`
+    - Blob sync failure after parse success -> mark source `failed` with `failureStage: "storage_sync"`
+  - Retry behavior:
+    - parse failure retries parse/upload
+    - `storage_sync` failure resumes sync from existing `documentId/jobId/revisionKey`, no reparse
+  - Existing remote Knowhere documents are readable immediately:
+    - SDK reads remote if Blob snapshot is missing/stale
+    - Blob sync is non-blocking
+    - missing Blob data must not produce zero chunks or non-ready behavior
+  - User-visible asset URLs must be durable Notebook Blob URLs:
+    - `readChunks` display calls use `assetUrlPolicy: "durable"`
+    - only visible/cited assets are hardened synchronously
+    - grep uses no asset URLs
+    - if hardening fails, return text and omit the asset URL rather than exposing a presigned Knowhere URL
+
+  ## CLI And MCP
+  - CLI/MCP configure disk `ParsedDocumentStorage` explicitly.
+  - CLI removes duplicate `documentId -> listChunks -> importJobResult` logic and delegates reads to SDK.
+  - MCP continues delegating to SDK `readChunks`, `grepChunks`, and `getDocumentOutline`; update descriptions/tests for storage-first remote-fallback behavior.
+  - Remote reads must not silently write local disk unless disk storage was explicitly configured.
+
+  ## Database And Compatibility
+  - Add a new Notebook migration only; do not edit existing migrations.
+  - Store snapshot/cache metadata in `source_parse_results`, including at least:
+    - `revision_key`
+    - `sync_status` or equivalent
+    - `failure_stage` / sync failure detail where appropriate
+    - progress cursor for resumable sync if not stored elsewhere
+  - Keep existing snapshot manifest shape compatible where possible, but treat committed manifest as the only readable snapshot marker.
+
+  ## Test Plan
+  - SDK:
+    - storage hit reads without remote calls
+    - stale/missing storage falls back to remote
+    - remote reads schedule non-blocking sync
+    - parse writes configured storage before completion
+    - paged `readChunks` behavior and invalid param combinations
+    - single-document bounded remote grep with truncation cursor
+    - remote outline fallback and storage outline path
+    - no implicit disk writes without configured disk storage
+  - Notebook:
+    - parse remains `parsing` until Blob sync completes
+    - parse success plus Blob sync failure marks `failed` with `storage_sync`
+    - retry of `storage_sync` resumes sync without reparsing
+    - existing remote document with no Blob snapshot returns chunks from remote
+    - missing/stale Blob never returns zero chunks for ready remote docs
+    - displayed/chat citation assets are Blob URLs or omitted, never Knowhere presigned URLs
+  - CLI/MCP:
+    - remote document read/grep/outline works through SDK
+    - disk storage is explicit
+    - duplicate remote-to-local import path is removed
+  - Verification:
+    - run targeted SDK storage/read tests
+    - run Notebook parse/sync/chunk/chat tests
+    - run CLI read/inspect tests and MCP tests
+    - run feasible typecheck/lint/test suites per repo
+    - smoke test local Knowhere + Notebook: parse PDF, wait until ready, inspect Blob manifest/pages/assets, open chunks, chat with citation, verify durable Blob URLs
+
+  ## Assumptions
+  - `documents.listChunks` is the authoritative remote parsed read API for published parsed v2 documents.
+  - `jobResultId ?? jobId` is sufficient as the revision key.
+  - Full job-result ZIP loading is allowed for explicit import/sync workflows, but not for Notebook request-time reads.
+  - Notebook request-time work is bounded to the requested page/window plus visible/cited asset hardening.

package.json

@@ -39,7 +39,7 @@
     "LICENSE"
   ],
   "engines": {
-    "node": ">=20.19.0",
+    "node": ">=22.13.0",
     "npm": ">=10.0.0",
     "pnpm": ">=9.0.0"
   },

packages/mcp/README.md

@@ -19,9 +19,11 @@ npx -y @ontos-ai/knowhere-mcp login
 npx -y @ontos-ai/knowhere-mcp
 ```
 
-The server uses stdio transport and stores expanded Knowhere result files under
-the SDK local knowledge cache by default. `knowhere-mcp login` opens the
-Knowhere dashboard in your browser and stores a local MCP login at
+The server uses stdio transport. Parse tools store expanded Knowhere result
+files under the SDK local knowledge cache; published `documentId` reads use the
+SDK's remote chunk fallback unless the host explicitly configures a cache
+directory for parsed-storage snapshots. `knowhere-mcp login` opens the Knowhere
+dashboard in your browser and stores a local MCP login at
 `~/.knowhere-node-sdk/mcp/auth.json`.
 
 During login, the dashboard asks for a Permission:
@@ -196,18 +198,23 @@ When logged in with Read only permission, the MCP server exposes only
   passed to outline/read/grep tools.
 - `knowhere_delete_document`: archive, or soft-delete, a published Knowhere
   document through the Knowhere API.
-- `knowhere_get_document_outline`: inspect a parsed document outline by passing
-  `localDocumentId`, published `documentId`, or completed `jobId`.
-- `knowhere_read_chunks`: read exact chunks from a parsed document by passing
-  `localDocumentId`, published `documentId`, or completed `jobId`.
-- `knowhere_grep_chunks`: run literal or regex grep over parsed document chunks
-  by passing `localDocumentId`, published `documentId`, or completed `jobId`.
+- `knowhere_get_document_outline`: inspect one parsed document outline by
+  passing `localDocumentId`, published `documentId`, or completed `jobId`.
+- `knowhere_read_chunks`: read exact chunks from one parsed document by passing
+  `localDocumentId`, published `documentId`, or completed `jobId`. Use
+  `page`/`pageSize` for display reads; `assetUrlPolicy: "durable"` returns only
+  storage-hardened asset URLs.
+- `knowhere_grep_chunks`: run literal or regex grep over one parsed document by
+  passing `localDocumentId`, published `documentId`, or completed `jobId`.
+  Broad workspace search belongs to `knowhere_search`.
 - `knowhere_search`: search published documents through the Knowhere API
   retrieval query.
 
-Read-related tool responses include `document.resultDirectoryPath`; expanded
+Local-cache tool responses include `document.resultDirectoryPath`; expanded
 chunks are stored at `<resultDirectoryPath>/chunks.json` for direct filesystem
-reads when needed.
+reads when a local cache entry exists. Remote fallback responses use
+`remote:<documentId>` or `parsed-storage:<documentId>` markers instead of
+claiming a local expanded result directory.
 
 ## Package Boundary