Add linked project knowledge vault to Memory

docs/changes/unreleased/1943-project-knowledge.md

@@ -0,0 +1,19 @@
+---
+kind: added
+title: Browse and maintain linked project knowledge from Memory
+pr: 1943
+surface: [chat, engine, docs]
+invalidates: []
+---
+
+Memory offers an opt-in project knowledge vault with linked Markdown articles,
+folder navigation, search, readable source quotations, and a knowledge model
+setting. Saved chat turns support incremental capture and sourced agent recall.
+
+Edit page Markdown or confirm a deletion from the reader. Human edits survive
+background compilation and are distinguished from original source history.
+Deleted canonical pages stay out of browsing and recall; original chats and
+related pages remain. Saved memory stays available for concise standing context.
+
+Knowledge compilation is experimental. Editing and deletion publish locally
+without model calls; deleting a page is not semantic erasure from source chats.

experiments/knowledge/BINARY-INTEGRATION.md

@@ -0,0 +1,153 @@
+# Memory and connected knowledge in the experimental binary
+
+Checkpoint: `219588e` (inline links); previous complete workflow: `77fba96`.
+This branch implements the local persisted-chat integration described below.
+
+## User experience
+
+Keep the top-level tab named **Memory**. Offer **Knowledge** and **Saved memory**
+within it. A new profile has no knowledge capture until the person chooses
+**Create knowledge base**. Explain that this starts collecting future chats;
+importing old conversations is a separate explicit action.
+
+Knowledge uses a folder/topic tree beside a readable article. Inline topic
+references open canonical pages; the person can return through navigation
+history, inspect incoming/outgoing links, search, and read source quotations.
+The tree must adapt to narrow terminals. Keep normal place navigation and avoid
+stealing printable characters from search. Show empty, pending, updating, failed,
+and ready states honestly. Existing saved-memory controls remain reachable.
+
+## Why keep both
+
+Saved memory already admits short scoped preferences, explicit pins and binding
+rules to chat context. Knowledge stores subject explanations, relationships,
+open questions, disagreements and source evidence. A useful article is not
+automatically a standing instruction. Preserve both stores and their semantics;
+retrieve knowledge when relevant rather than inject the whole vault each turn.
+
+The experimental binary now captures opted-in local chats and exposes cited
+knowledge tools. This does not install or enable knowledge in the default binary.
+
+## Implementation constraints
+
+Reuse the tested Python compiler through an embedded, versioned runtime adapter;
+do not require the source checkout at runtime. Check Python 3 availability.
+Capture completed durable journal records with stable source IDs and recoverable
+cursors. Human statements and agent synthesis retain different provenance.
+In `userMessage`, `authored` means session-authored, not human-authored. Never
+capture synthetic digests as words the person typed.
+
+Scope every vault, capture and retrieval by authenticated profile/project.
+Do not merge concurrent unrelated projects or expose local knowledge under a
+remote host's identity. Keep network and compilation work off the TUI update
+loop and agent mutex. Persist queued data before attempting model work.
+
+Offer retrieval-only search/read tools on both chat tool belts. Return bounded
+page text and source references; let the current chat model formulate the
+answer. An additional model answer-generation request is unnecessary.
+
+## Scheduling and cost
+
+Queue committed content locally with zero model requests. Combine eligible
+events from the same partition after 30 seconds quiet or 120 seconds maximum
+age, bounded by the compiler's source/context caps. Process remaining events on
+later passes. Unchanged content requires zero calls. New content gets bounded
+extraction and reconciliation; full guide rewrites and curiosity maintenance
+remain explicit until their runtime budgets are demonstrated.
+
+All live model roles must use OpenRouter `deepseek/deepseek-v4.1-flash` and verify
+the returned model. Credentials stay in the existing private credential path or
+child environment, never command arguments, logs or source. Record extraction,
+composition, retries and failures separately. A local link/search action costs
+zero model requests. Do not use the OpenRouter Batch API.
+
+## Acceptance and rollout
+
+Approved test interfaces: Memory actions to vault creation and linked
+navigation; committed conversations to incremental capture and cited retrieval.
+Earlier approved extraction/recall contracts continue to apply.
+
+Validate no capture before opt-in, durable retry without duplicate publication,
+project isolation, self/ambiguous/missing-link handling, keyboard navigation,
+and retention of ordinary saved memory. Verify actual live TUI plus chat
+retrieval with the pinned model, record usage, and review against the checkpoint.
+Build a separate experimental binary; preserve the installed default.
+Keep recordings and generated knowledge outside source control.
+
+## Implemented limits
+
+The adapter needs Python 3 on Linux or macOS. Compilation runs while a persisted
+local chat is active; pending journals recover when their chat reopens. There is
+no always-on global service scanning dormant chats. New guide discovery is
+bounded; existing guide prose is retained, with newly captured member claims
+exposed under sourced **Recent knowledge** until editorial integration. This
+avoids full-guide model rewrites on each turn. Superseded support is withheld.
+
+Normal and bash-belt chats plus their session child workers can retrieve pages.
+Standalone `codeaf do`, remote knowledge transport and the desktop are not wired
+by this experiment. Hosted windows explicitly refuse local-vault fallback.
+Tool responses are capped at 12,000 bytes; long reads provide versioned cursors.
+
+Knowledge search/read make no extra model calls. Their small tool schemas remain
+in the persisted chat's cached tool belt before opt-in so creation becomes usable
+without restarting; those schema tokens are part of ordinary chat cost. The
+metadata-only status poll does not hydrate articles. Imported sources are archived
+by identity, indexed pending counts avoid rescanning their contents, and unchanged
+projections skip export. Vault JSON hydration and lexical search still grow with
+collection size; large-scale throughput and long-running quality remain work.
+
+## Reader polish checkpoint
+
+The reader uses the chat UI's underlined link treatment for `[[topic]]` references,
+with the same canonical target for mouse clicks and the Ctrl+L link list. Tree
+selection uses the shared cursor ground, with a textual marker on terminals that
+cannot paint it. Leaf rows omit decorative diamonds. Paragraphs wrap at at most
+100 cells, while narrow terminals keep the existing tree/article navigation.
+
+Sources are a compact clickable disclosure (Ctrl+O), grouped by original source,
+with readable speaker/date and quotations. Return restores the article's reading
+position; source identities remain in the vault rather than repeated in prose.
+The reader changes require no extraction, reconciliation, or model requests;
+incremental model cost for this UI iteration is $0. Infrastructure and ordinary
+chat/extraction costs are separate.
+
+Snapshot adoption builds page and connection indexes. Steady frames format only
+visible tree rows; hover reads the displayed frame's hit map. A synthetic
+120×40-terminal benchmark with short articles and one edge per page measured
+about 55 µs/frame at 1,000 pages and 46 µs at 10,000 pages (sampling noise; neither
+includes terminal/network delivery). All-match filtering measured 0.49 ms and
+4.82 ms respectively. Filtering and snapshot hydration still grow with corpus
+size, and dense links or long articles need separate measurements. These are
+layout measurements, not end-to-end SSH latency guarantees. Remote `--host`
+knowledge transport remains unsupported; ordinary SSH runs the vault reader on
+the machine where the binary executes.
+
+## Sidebar interaction follow-up
+
+The wheel uses its pointer coordinates: over the sidebar it advances the tree
+viewport immediately while retaining the article; over the article it scrolls
+only the article (or the connected-page list). Wheel handling reads cached frame
+hits, so it neither reloads the vault nor renders a frame to discover the pane.
+Collapse all / Expand all are clickable, hoverable controls; Ctrl+Left /
+Ctrl+Right provide keyboard access. They clear search, preserve the current
+article, and fold nested folders consistently. Folding remains an explicit
+whole-tree operation. These interactions require no model requests.
+
+## Sidebar, sources and model settings
+
+Wide readers keep the Knowledge heading, page metadata, saved-memory switch,
+search and folder actions in a fixed left column. Article content begins at the
+top of the right column. Both memory views expose clickable, hoverable switches;
+Alt+U remains available. Source excerpts use the shared Markdown renderer without
+rewriting their original stored text. One source projection is cached per note,
+width and measured prose style; snapshots invalidate it. A synthetic 100-source
+warm-frame benchmark measured 5.36 ms / 4.07 MB before caching and 0.059 ms /
+42 KB after, excluding terminal and network delivery.
+
+Settings → Models → knowledge configures background compilation,
+separately from chat/task model choices. Only the OpenRouter catalog is offered.
+The default remains deepseek/deepseek-v4.1-flash; a saved change takes effect at
+the next compilation, including every compiler role, without changing existing
+notes. Provider receipts are checked against the chosen model, and caches are
+scoped by model. This iteration's tests use offline provider fixtures: additional
+model/API charges are $0, not a new production cost estimate.

experiments/knowledge/CRITICAL-REVIEW.md

@@ -0,0 +1,69 @@
+# Critical review: useful pages are not yet a dependable knowledge base
+
+The current experiment demonstrates sourced subject pages and bounded recall. It
+has not demonstrated complete capture, primary-source accuracy, stable long-term
+organization, or production cost at sustained load. Model acceptance and article
+length are not quality metrics.
+
+## Findings and changes
+
+| Finding | Change in this iteration | Remaining limit |
+|---|---|---|
+| Every guide paragraph depended on its whole subject | Depend only on cited claims; keep exact raw-source hashes separately | Uncited semantic dependencies and later conflicting new evidence still need discovery |
+| Source citations can lead back to an agent's own summary | Project source roles, claim kinds and unresolved conflict flags onto prose; expose qualification in the inspector | Role is coarse provenance, not factual verification; tool fetch versus tool write needs explicit host metadata. independently_verified stays false because this experiment performs no independent verification |
+| Sourced prose can still be a temporary execution diary | Separate evidence support and lasting reference usefulness gates; both are required | Same-model judgements can miss noise; historical present-tense descriptions still need freshness checks |
+| Review repeats quotes and includes uncited records | Send cited claims/blocks once and use block IDs in paragraphs | Same-model semantic review remains fallible |
+| Conflict awareness arrived after writing | Supply known conflict candidates to both writer and reviewer | Candidate detection is incomplete; no automatic adjudication or external verification |
+| Quantity can disguise weak organization | Audit question saturation, missing topic links and unsupported reading paths | Hierarchy/identity quality needs held-out and longitudinal evaluation |
+
+Source role/date changes invalidate learner reuse. Conflict changes currently replan the whole guide; paragraph-level cached composition is not yet implemented. Retired-claim conflict pairs remain in history, while current prose badges only include pairs with active alternatives.
+
+The guide dependency regression exercises the existing sourced-note seam: retire
+an unrelated claim and the paragraph survives; retire its actual cited support and
+it disappears. Raw-only paragraphs remain dependent on their source hashes; this
+does not mean future contradictions can be detected without new evidence intake.
+
+## Next experiments, in order
+
+1. **Evidence first.** Host-recorded source types distinguish user intent,
+   repository snapshots, primary document fetches, command results and agent
+   synthesis. An agent-written dossier with URLs remains a synthesis until those
+   sources are actually retrieved. Keep valid-at and recorded-at times distinct.
+   Measure incorrect promotion and unresolved contradiction handling on held-out
+   cases; do not infer independence from source IDs or repeated wording.
+2. **Stable topic maps.** Separate durable concept IDs from titles/folders. Review
+   split/merge suggestions as reversible changes with redirect/history records.
+   Build maps of content from semantic memberships and evidence-linked relations;
+   do not invent relationship facts from co-occurrence. Measure disconnected
+   subjects, useful route length, duplicate coverage and churn after new chats.
+3. **Useful curiosity.** Ask fewer questions. Keep them only when missing knowledge
+   blocks understanding or a real task; retain why it matters, inspected evidence,
+   last attempt and wake condition. Rank by task relevance and topic coverage,
+   rather than number of claims or easily answered trivia. New evidence can wake a
+   parked question; unchanged time alone should not restart a failed search.
+4. **Push capture with selective repair.** Stage optional same-completion proposals,
+   deduplicate locally, debounce dirty updates across already permitted chats into
+   one ordinary bounded request. Use source->claim->paragraph reverse dependencies,
+   owner-scoped publication and idempotent event receipts. Maintain indexed source
+   postings to notice relevant unseen evidence without invalidating every subject
+   on every event. These production hooks are not installed yet.
+5. **Quality per token.** Freeze a held-out task set with expected original evidence,
+   corrections, negations, copied reports, sparse subjects and topic renames.
+   Compare capture omissions, retrieval support, answer correctness, changed-note
+   churn and total charged tokens. Use random omission audits as well as optional
+   proposals; neither self-selection nor a same-model judge proves recall.
+
+## Data direction
+
+SQLite remains sufficient for the experiment. Store immutable source events,
+versioned assertions, assertion/evidence support edges, paragraph dependencies,
+concept identities/redirects and dirty-work receipts. A subject page is a reading
+projection; Markdown is its portable export. Keep containment, related navigation,
+factual dependencies, contradictions and hypothesized connections as different
+edge types. A filesystem folder is one view, not the identity of the knowledge.
+
+Production decisions need measured throughput and concurrency first. No graph
+server, vector service, autonomous web-research loop or nightly full rewrite is
+justified solely by the existing page count. Designer-facing opportunities include
+source-basis badges, competing views, why-linked explanations, reading paths and
+staleness indicators; none should imply complete or independently verified truth.