Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions docs/design/brain/knowledge-structure.md
Original file line number Diff line number Diff line change
Expand Up @@ -875,3 +875,79 @@ The five layers, the entity page templates, and the backend interface are the *i
| L4 master index | `<knowledge>/index.md` | wiki-rebuild CLI, dream cycle | Kit's system prompt, humans |

Concepts, layers, formats — pinned. Backends, bots, and CLIs — swappable. Obsidian and Karpathy both happy. Engram or SQLite or LanceDB plug in later without rewriting any of the above.

---

## Addendum (2026-06-25): what shipped vs. this design

> Everything above is the original 2026-04-21 design and is kept as the design
> record, not patched in place. Where it and this addendum disagree, **the
> addendum is the current reality.** One thing moved: where the ontology lives.

The shape held. The five layers, the entity-page templates, eager entity
creation, and the Obsidian + backend-agnostic contracts all shipped roughly as
written. What changed is the **home of the ontology**.

### Original framing

`ontology.toml` was *"the static schema … loaded once, referenced everywhere"*
(see **Vocabulary — the static schema**, and the **Summary** table, which lists
Topic/Type/Person *schema* as living there). The implication: a central,
hand-maintained file is the source of truth for what entities exist and how
they relate.

### Current reality

The **ontology of record is the entity graph in the vault** — one `.md` per
entity (`kind: person | correspondent | topic | asset | story`), frontmatter as
identity, `[[wikilinks]]` as edges. Persons, correspondents, and topics are
generated as entity pages today; correspondents auto-create as leaf pages with
deterministic backlinks.

`ontology.toml` did **not** disappear, but it **demoted** to a single job: the
**classifier vocabulary seed** (topics, doctypes, synonyms, aliases) that the
capture and query-rewrite prompts read as an `ontology_section` (via
`stack.ontology.Ontology`, loaded from the vault or the shipped seed). It no
longer holds entity identity or the relation graph — the vault does.

So the **Summary** table above now reads:

| Concept | Original doc says | Current reality |
|---|---|---|
| Entity identity + relations | schema in `ontology.toml` | **vault entity pages** (`<bucket>/<id>.md`, `kind:` frontmatter, `[[links]]`) |
| Tagging vocabulary (topics, doctypes, synonyms) | `ontology.toml` | `ontology.toml` — unchanged, but now its *only* job |
| Durable facts | `facts.toml` / `facts.jsonl` | unchanged |

### Identity vs. overview, on one page

A consequence worth pinning: an entity page splits by **mutability**.
Frontmatter is the **durable identity** (slots, aliases, relations) and is never
regenerated; the body is the **current overview** (LLM, batch-regenerated,
recency-weighted). That split is the guard against recency-weighted prose
washing out stable facts — the stable facts sit in frontmatter, which the regen
never touches. Durable facts about an entity live on *that entity's* page;
other entities point at them with `[[links]]` rather than copying them.

### Why we moved this way

1. **Obsidian-native, no second source of truth.** Entity pages give backlinks
and graph view for free. A central registry has to be kept in sync with the
vault it describes; the vault describing itself removes that class of drift.
2. **One source per fact.** A fact lives on the entity it belongs to (the bus's
specs on the bus's page); relations are edges. A central ontology would
duplicate facts and force reconciliation.
3. **It auto-extends; a hand-file doesn't.** A new topic or correspondent
becomes a page on first sighting. A hand-maintained `ontology.toml` is a
manual gate, and in practice it stopped being maintained — which was the
signal that the entity graph had already become the real ontology.
4. **Portability.** A vault of OKF entity pages opens in any Obsidian/OKF tool
as a graph. A central proprietary schema does not travel.

### What this leaves open

Vocabulary (`ontology.toml`) and the entity graph (the vault) are still **two
stores**. The logical next step — the *living-ontology loader* in
[wiki-improvements-backlog.md](wiki-improvements-backlog.md) — would derive the
classifier vocabulary *from* the entity pages and tags-in-use, collapsing the
two into one. It is specced, not built. Until it lands, `ontology.toml` stays
the vocabulary seed and the entity graph stays the ontology of record.
1 change: 1 addition & 0 deletions stacklets/docs/bot/capture_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -725,6 +725,7 @@ async def _publish(
tags=tags,
existing_path=existing_path,
capture_id=capture_id,
submitter=sender_mxid,
)

# Feed topic tags (not the derived Person: X) back into the
Expand Down
77 changes: 75 additions & 2 deletions stacklets/docs/bot/git_mirror.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,29 @@

BOT_USERNAME = "archivist-bot"
BOT_EMAIL = "archivist-bot@local"


def _commit_author(submitter: str | None) -> tuple[str, str]:
"""`(author_name, author_email)` for a capture commit.

Attributes the commit to the family member who filed it, derived from
their Matrix id (`@marge:merles.eu` -> `marge`, `marge@merles.eu`), so
`git log --author` answers "who added this". Falls back to the bot when
there is no submitter or the id is malformed — the commit still lands.
"""
if submitter:
local = submitter.split(":")[0].lstrip("@").strip().lower()
domain = submitter.split(":", 1)[1].strip() if ":" in submitter else ""
if local:
return local, f"{local}@{domain}" if domain else f"{local}@local"
return BOT_USERNAME, BOT_EMAIL


def _filer_localpart(submitter: str | None) -> str | None:
"""The submitter's Matrix localpart, for the `filed_by` frontmatter field."""
if not submitter:
return None
return submitter.split(":")[0].lstrip("@").strip().lower() or None
TOKEN_NAME = "archivist-git-mirror"
TOKEN_SCOPES = ["write:repository", "read:repository", "read:user"]

Expand Down Expand Up @@ -295,6 +318,41 @@ async def _lookup_path(self, client: ForgejoClient, paperless_id: int) -> str |
return path
return None

async def _lookup_capture_path(
self, client: ForgejoClient, target_path: str,
) -> str | None:
"""An existing capture with `target_path`'s identity, ignoring its title.

A capture's filename is `<title-slug>-<hash>.md`; the hash is stable
per URL/body, the slug is not. So the same link re-pasted with different
framing, or content re-classified to a different title, would otherwise
fork a near-duplicate. Match on the `<entity>/<kind>s/` folder + the
`-<hash>.md` suffix instead: return the exact path when it exists
(in-place update), else a same-hash file under a different slug (so the
caller renames it), else None.
"""
parts = target_path.split("/")
if len(parts) < 2:
return None
prefix = f"{parts[0]}/{parts[1]}/" # <entity>/<kind>s/
suffix = "-" + target_path.rsplit("-", 1)[-1] # -<hash>.md
try:
tree = await asyncio.to_thread(
client.list_tree, self.repo_owner, REPO_NAME,
)
except ForgejoError:
return None
match = None
for entry in tree:
path = entry.get("path", "")
if entry.get("type") != "blob":
continue
if path.startswith(prefix) and path.endswith(suffix):
if path == target_path:
return path # exact identity + title → update in place
match = path # same identity, different title → rename
return match

# ── Filename, frontmatter, body ──────────────────────────────────────

def _slug(self, text: str) -> str:
Expand Down Expand Up @@ -605,12 +663,13 @@ def _capture_frontmatter(
tags: list[str],
model: str | None,
capture_id: str | None = None,
filed_by: str | None = None,
) -> dict:
"""Capture frontmatter. Delegates to ``vault_entry.capture_frontmatter``."""
return capture_frontmatter(
title=title, captured_at=captured_at, kind=kind,
source_uri=source_uri, persons=persons, tags=tags,
model=model, capture_id=capture_id,
model=model, capture_id=capture_id, filed_by=filed_by,
)

async def read_capture(self, path: str) -> str | None:
Expand Down Expand Up @@ -648,6 +707,7 @@ async def publish_capture(
tags: list[str] | None = None,
existing_path: str | None = None,
capture_id: str | None = None,
submitter: str | None = None,
) -> str | None:
"""Create or update a capture entry in the mirror.

Expand Down Expand Up @@ -698,6 +758,14 @@ async def publish_capture(
hash_key=hash_key,
)

# Dedup on identity, not filename: if this URL/body already lives under
# a different title (a re-paste with new framing, an older differently-
# titled capture), find it by its hash so we update/rename that entry
# instead of forking a duplicate. The reprocess caller already knows the
# path, so only look when it didn't pass one.
if existing_path is None:
existing_path = await self._lookup_capture_path(client, target_path)

# Reprocess path: read the previous file at its old path. When
# the title changes the slug changes too, so we'll delete the
# old entry after writing the new one (same as the doc
Expand All @@ -717,6 +785,10 @@ async def publish_capture(
)
existing_at_new = existing if lookup_path == target_path else None

# Attribute the capture to whoever filed it: the localpart goes in
# `filed_by` frontmatter, and the same identity becomes the git commit
# author (committer stays the bot). Both come from `submitter`.
author_name, author_email = _commit_author(submitter)
fm = self._capture_frontmatter(
title=title,
captured_at=captured_at,
Expand All @@ -726,6 +798,7 @@ async def publish_capture(
tags=tags or [],
model=model,
capture_id=capture_id,
filed_by=_filer_localpart(submitter),
)

briefing_summary = classification.get("summary")
Expand Down Expand Up @@ -762,7 +835,7 @@ async def publish_capture(
self.repo_owner, REPO_NAME, target_path,
content=content, message=message,
sha=existing_at_new["sha"] if existing_at_new else None,
author_name=BOT_USERNAME, author_email=BOT_EMAIL,
author_name=author_name, author_email=author_email,
)
# Title rename on reprocess: remove the prior file after
# the new one is in place so the vault never has both.
Expand Down
14 changes: 13 additions & 1 deletion stacklets/docs/bot/pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
import json
import re
from dataclasses import dataclass, field
from datetime import date
from typing import TYPE_CHECKING, Any

import aiohttp
Expand Down Expand Up @@ -575,6 +576,7 @@ async def classify(

response = await self._llm.complete(
"classifier", prompt, images=attach, json_mode=True,
temperature=0.0,
)
if not response:
return {}
Expand Down Expand Up @@ -620,6 +622,7 @@ async def classify_capture(
existing_tags=existing_tags or [],
user_hint=user_hint,
initial_classification=initial_classification,
today=date.today().isoformat(),
)
valid_images = [
img for img in (images or [])
Expand All @@ -635,6 +638,7 @@ async def classify_capture(
)
response = await self._llm.complete(
"classifier", prompt, images=attach, json_mode=True,
temperature=0.0,
)
if not response:
return {}
Expand Down Expand Up @@ -978,6 +982,7 @@ def _build_capture_prompt(
existing_tags: list[str] | None = None,
user_hint: str | None = None,
initial_classification: dict | None = None,
today: str | None = None,
) -> str:
"""The capture prompt — smaller and focused on summary + tags.

Expand All @@ -995,6 +1000,13 @@ def _build_capture_prompt(
synonyms across the whole corpus later.
"""
existing_tags = existing_tags or []
# The model needs an anchor to turn relative-time phrases ("this Friday",
# "the 14th to 16th") in a note into concrete dates in the facts.
today_line = (
f'\nToday\'s date is {today}. Use it to resolve relative-time phrases '
f'("this Friday", "next week", "the 14th to 16th") into concrete dates.\n'
if today else ""
)
tags_hint = (
f"Existing tags in use: {json.dumps(existing_tags, ensure_ascii=False)}\n"
"Prefer these when they fit. Only invent new tags when nothing existing matches.\n"
Expand All @@ -1005,7 +1017,7 @@ def _build_capture_prompt(

return f"""Summarize and tag this content for a personal knowledge vault.
Return ONLY a JSON object.

{today_line}
The user is bookmarking or noting this content to find it later. Your
job: produce a digest they can scan in 10 seconds and tags that
position this content among their interests.{_initial_classification_block(initial_classification)}{_user_hint_block(user_hint)}
Expand Down
6 changes: 6 additions & 0 deletions stacklets/docs/bot/vault_entry.py
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,7 @@ def capture_frontmatter(
tags: list[str],
model: str | None,
capture_id: str | None = None,
filed_by: str | None = None,
) -> dict:
"""Frontmatter for a capture entry.

Expand Down Expand Up @@ -322,6 +323,11 @@ def capture_frontmatter(
fm["date"] = captured_at
if persons:
fm["persons"] = persons
if filed_by:
# The Matrix localpart of whoever filed this capture. Mirrors the git
# commit author so "who added this" survives outside a git context
# (Dataview queries, a plain read of the file).
fm["filed_by"] = filed_by
if tags:
fm["tags"] = tags
if source_uri:
Expand Down
Loading
Loading