diff --git a/README.md b/README.md index 1a736ce..c49e284 100644 --- a/README.md +++ b/README.md @@ -44,15 +44,25 @@ The reference implementation runs on a Mac Studio M1 in our living room at Lake - **Private family chat** with native iOS, Android, macOS, Windows and web apps (Matrix + Element X). - **Photo library and mobile backup** that replaces iCloud Photos for the whole family (Immich). - **Document archive with OCR** that you photograph from your phone and the local AI files for you (Paperless-ngx). +- **Your family wiki**: an Obsidian-compatible second brain, generated from the documents, notes, and voice memos you file (memory). - **Local AI engine** on Apple Metal GPU: voice transcription, text-to-speech, document classification (oMLX + Whisper + Piper). - **A bot runtime in chat** that automates the small stuff: filing receipts, transcribing voice memos, status reports. - **One CLI to operate it all**: `./stack up ` and it is running. -

- -

- -

The archivist files and finds documents from your phone, in chat

+ + + + + + + + + +
+ + + Homer asks in the family chat whether driving under the influence of Duff beer is covered by the car insurance. The archivist answers no, citing the policy documents. +
The archivist files and finds documents from your phone, in chatAsk the wiki a question in chat, get an answer with sources
Everything runs on your Mac. Nothing leaves your network unless you tell it to. diff --git a/docs/admin-guide.md b/docs/admin-guide.md index ce8271d..6c0357c 100644 --- a/docs/admin-guide.md +++ b/docs/admin-guide.md @@ -134,7 +134,10 @@ famstack uses the 420xx range to avoid conflicts with anything you might already | 42040 | code (Forgejo) | | 42050 | chatai (Open WebUI) | | 42060 | ai (oMLX) | +| 42061 | ai (LM Studio, optional GGUF backend) | | 42062 | ai (Whisper) | +| 42063 | ai (Piper TTS) | +| 42070 | memory (family wiki) | If a port is taken, stop the offender or switch to `domain` mode (see [stack-reference.md](stack-reference.md)). @@ -425,7 +428,7 @@ The persistent layer of the household brain: a curated knowledge vault, rendered Setup seeds the vault with three things: the classification ontology (the topics and document types the archivist files against), household facts, and a hand-curated correspondents layer with aliases, so "Springfield Insurance" and "Springfield Ins. Co." resolve to the same page instead of becoming duplicates. -The wiki rebuilds itself when the vault changes: file a document in chat, refresh the page. Edits happen in Forgejo (every wiki page links to its source), so the commit log doubles as the household's learning history. +The wiki maintains itself. A curator sidecar (`stack-memory-curator`) watches the vault: when new filings settle it regenerates the pages of the family members involved plus the home page (a couple of LLM calls, a few minutes after the burst), and once a night it rebuilds everything — topic pages, cross-references, the lot — while the GPU has nothing better to do. `./stack memory wiki` stays available as the manual trigger, and `[memory]` in `stack.toml` holds the knobs (see [Configuration](#configuration)). Edits happen in Forgejo (every wiki page links to its source), so the commit log doubles as the household's learning history. | | | |---|---| @@ -465,6 +468,11 @@ schedule = "0 0 3 * * *" # Watchtower nightly image updates [ai] default = "mlx-community/Qwen3.5-9B-MLX-4bit" # change to match your RAM language = "en" # "de" for German voice/transcription + +[memory] +wiki_auto_rebuild = true # curator refreshes member pages after filings +wiki_rebuild_quiet_secs = 180 # how long the vault must stay quiet first +wiki_nightly = "03:30" # nightly full rebuild, local time ("" disables) ``` Key things to know: diff --git a/docs/agent/ops.md b/docs/agent/ops.md index 46de4b3..a4b2c30 100644 --- a/docs/agent/ops.md +++ b/docs/agent/ops.md @@ -45,6 +45,8 @@ If a precondition is missing, `./stack` prints exactly what to do. Don't improvi **Output contract:** every command returns JSON when piped or when `--json` is passed. Force human output with `--pretty`. Exit code 0 == success. +**Applying config/compose changes to a running stacklet:** `stack up ` is the apply command — it re-renders `.env` and `compose up` creates new services and recreates changed ones, leaving the rest running. `restart` (= `down` + `up`) also works but takes the whole stacklet down first. Plain `docker compose restart` does neither: it won't re-render env or create new services. + ## Invariants - `stack up` is **always safe to re-run.** Failures leave the system in a valid intermediate state; re-running picks up where it failed. @@ -82,6 +84,7 @@ Refuse without explicit, scoped human approval: | 42050 | chatai | Open WebUI | | 42060 | ai | oMLX | | 42062 | ai | Whisper | +| 42070 | memory | Family wiki (Quartz) | Port collisions: do not silently rebind. Surface them. The user's fix is "stop the offender" or switch to domain mode. @@ -99,6 +102,9 @@ Port collisions: do not silently rebind. Surface them. The user's fix is "stop t | LLM OOM / very slow | RAM tier | Edit `[ai] default` to smaller model, then `./stack setup ai`. | | Disk full | `./stack host` | Likely the photo library. Move `data_dir` to external SSD. | | Element warns "browser not supported" | n/a | Click Continue. The check is outdated; Element works in every modern browser. | +| Wiki pages stale after filings | `docker logs stack-memory-curator` | Curator debounces (~3 min quiet) before rebuilding; topic pages wait for the nightly sweep. Manual override: `./stack memory wiki`. | +| Wiki shows old content entirely | `docker logs stack-memory-curator` | The curator owns the vault `git pull` (the wiki container is a read-only view). Curator down/stuck → vault and wiki go stale together. | +| Curator logs "waiting for vault" | `./stack status` → memory, code | Vault not cloned yet — the memory install hooks own the initial clone. | For symptoms not on this table: `./stack logs ` + `./stack errors`, paste output to the user. **Do not invent fixes.** diff --git a/docs/assets/memory-wiki-answer.png b/docs/assets/memory-wiki-answer.png new file mode 100644 index 0000000..4a59e71 Binary files /dev/null and b/docs/assets/memory-wiki-answer.png differ diff --git a/docs/cleanup-backlog.md b/docs/cleanup-backlog.md index eda1700..67d661b 100644 --- a/docs/cleanup-backlog.md +++ b/docs/cleanup-backlog.md @@ -3,6 +3,41 @@ Items land here with a reason. Surface them when adjacent code is touched (see `docs/agent/dev.md`, Pre-1.0 conventions). +## Wiki freshness follow-ups (curator shipped 2026-06-11) + +The curator sidecar ships the first two freshness tiers: debounced +incremental rebuilds (persons + home) and the nightly full sweep. +Design notes that survive it, for whoever touches this next: + +- **Realtime is NOT a requirement.** The mirror is realtime; the wiki + is a derived view. The nightly sweep makes the incremental person + mapping merely *helpful*, never load-bearing — worst case for a + mapping miss is "stale until tonight". Don't grow the incremental + heuristics; grow the deriver instead. +- **Page update strategy — design when it's time, but the tension is + known (2026-06-11):** full regeneration resamples page quality (a + good page can regress on the next sweep); evolving the existing + page accumulates errors that self-cite (the "Bartley [5]" finding). + Most promising middle: a fact-checking pass — "page + sources, fix + what the sources don't support, touch nothing else" — anchored to + ground truth while preserving good prose. Likely CLI shape then: + `wiki` = update/check, `wiki rebuild` = fresh full generation. +- **famstacker `wiki` command** (Server Room chat trigger) is the + missing third tier: "CLI commands are the primitives, the bot runs + or offers them". Needs the famstack API to allow the command and an + ack-then-report shape for the multi-minute run. +- `{"cmd": "notify"}` for the famstack API (containers → Server Room + via `stack messages send`) lives in the git stash + ("wiki auto-rebuild: curator sidecar + API notify") — platform + piece, ship it with whichever consumer arrives first; the curator's + completion notice is a natural one. +- Rejected runtime homes, don't re-litigate: host daemons (no launchd + surface), quartz container (node image; "the wiki never writes"), + bot-runner service concept (one consumer), bot-runner image reuse + (the curator uses 2 of its 10 deps; slim image won). + +## CI release gate (post-0.3) + ## CI release gate (post-0.3) The pre-tag gate in `docs/agent/dev.md` is manual; v0.3.0-beta.1 shipped diff --git a/lib/stack/ai/client.py b/lib/stack/ai/client.py index f56015a..24de4b6 100644 --- a/lib/stack/ai/client.py +++ b/lib/stack/ai/client.py @@ -208,13 +208,18 @@ def _full_role(self, role: str) -> str: async def complete(self, role: str, prompt: str, *, images: "list | None" = None, json_mode: bool = False, - model_override: str | None = None) -> str: + model_override: str | None = None, + temperature: float | None = None) -> str: """Run a single chat completion and return the response text. ``role`` resolves to a concrete model via `resolve_model`. Pass ``images`` (objects with ``.data``/``.mime``) for a multimodal - call, and ``json_mode=True`` to ask for a JSON object back. SDK - errors are translated to the typed LLM errors above. + call, and ``json_mode=True`` to ask for a JSON object back. + ``temperature`` pins sampling when output stability matters + (the wiki generator regenerates pages and wants the same + evidence to yield the same page); None keeps the server + default. SDK errors are translated to the typed LLM errors + above. """ model = model_override or resolve_model(self._full_role(role)) content = prompt if not images else _content_parts(prompt, images) @@ -222,6 +227,8 @@ async def complete(self, role: str, prompt: str, *, kwargs: dict = {} if json_mode: kwargs["response_format"] = {"type": "json_object"} + if temperature is not None: + kwargs["temperature"] = temperature try: resp = await self._client.chat.completions.create( diff --git a/lib/stack/stack.py b/lib/stack/stack.py index 15fc447..8ee2c4a 100644 --- a/lib/stack/stack.py +++ b/lib/stack/stack.py @@ -178,6 +178,11 @@ def _build_template_vars(self) -> dict: # Default "family" fits famstack; deskstack or non-family # deployments override to "office", "household", a surname, etc. "shared_bucket": self._cfg("core", "shared_bucket", "family"), + # Wiki freshness (the memory stacklet's curator sidecar). + # Bool/int from stack.toml render as env-friendly strings. + "wiki_auto_rebuild": str(self._cfg("memory", "wiki_auto_rebuild", True)).lower(), + "wiki_rebuild_quiet_secs": str(self._cfg("memory", "wiki_rebuild_quiet_secs", 180)), + "wiki_nightly": self._cfg("memory", "wiki_nightly", "03:30"), # AI service URLs — host-side for CLI, docker-side for containers "ai_openai_url": ai_openai_url, "ai_openai_url_docker": ai_openai_url.replace("://localhost", "://host.docker.internal"), diff --git a/stacklets/memory/Dockerfile b/stacklets/memory/Dockerfile index 381dbe6..876f88b 100644 --- a/stacklets/memory/Dockerfile +++ b/stacklets/memory/Dockerfile @@ -40,8 +40,8 @@ RUN git clone --depth 1 --branch "${QUARTZ_TAG}" https://github.com/jackyzha0/qu COPY quartz/quartz.config.ts ./quartz.config.ts COPY quartz/quartz.layout.ts ./quartz.layout.ts -# The entrypoint keeps the vault in sync (background git pull loop) and -# then serves it with Quartz. See quartz/entrypoint.sh. +# The entrypoint serves the vault with Quartz — a pure view. The +# curator sidecar owns the git pull. See quartz/entrypoint.sh. COPY quartz/entrypoint.sh /usr/local/bin/wiki-entrypoint.sh RUN chmod +x /usr/local/bin/wiki-entrypoint.sh diff --git a/stacklets/memory/bot/cli/wiki.py b/stacklets/memory/bot/cli/wiki.py index bbf0c5e..f0f4a6e 100644 --- a/stacklets/memory/bot/cli/wiki.py +++ b/stacklets/memory/bot/cli/wiki.py @@ -10,6 +10,13 @@ stack memory wiki --topics every topic page, no home/members stack memory wiki --dry-run preview to stdout, no writes +`--member` and `--topic` repeat and combine with `--home`: any +selection flag switches from the full sweep to "generate exactly this +set" — one invocation covers every page a filing burst touched (the +curator's incremental path). Member values take a slug or a display +name ("Homer Simpson" hits homer's bucket); unknown slugs warn and +skip, and the run fails only when nothing in the selection matched. + Pages are published to the memory repo on Forgejo, where the wiki container picks them up within seconds. The wiki is Quartz, which renders `index.md` as the landing page for the site (vault-root @@ -68,6 +75,19 @@ _TOKEN_NAME = "stack-memory-wiki-cli" _TOKEN_SCOPES = ["write:repository", "read:repository"] +# Subject prefix of every commit this command pushes. The curator's +# poll loop filters on it so the wiki's own publishes never trigger +# another rebuild — change it and that loop comes back. +COMMIT_PREFIX = "docs(memory): refresh" + +# Decoding temperature for page generation. Sampling turns every +# rebuild into a dice roll: the same evidence produced visibly +# different pages run to run (observed live 2026-06-11 — a rich facts +# list vanished on a one-note evidence change). Default 0 (greedy) so +# a page is a near-deterministic function of its evidence; the env +# override exists for experiments and unusual models. +_TEMPERATURE = float(os.environ.get("WIKI_TEMPERATURE", "0.0")) + # Bracketed regenerate markers. Match the correspondent-page convention # verbatim so the same human contract holds across every generated page: # edits outside the brackets are preserved, inside is ours to rewrite. @@ -122,10 +142,10 @@ def _extract_citations(text: str) -> list[int]: async def run(llm: LLM, argv: list[str]) -> int: dry_run = "--dry-run" in argv - home_only = "--home" in argv + home_sel = "--home" in argv topics_only = "--topics" in argv - single_member = _arg_value(argv, "--member") - single_topic = _arg_value(argv, "--topic") + members_sel = _arg_values(argv, "--member") + topics_sel = _arg_values(argv, "--topic") vault_dir = os.environ.get("MEMORY_VAULT_DIR", "") if not vault_dir: @@ -163,50 +183,85 @@ async def run(llm: LLM, argv: list[str]) -> int: vault, shared_bucket=shared_bucket, member_slugs=roster, ) - # ── Single-surface flags ────────────────────────────────────────── - # `--topic camping` finds the (bucket, slug) match anywhere in the - # vault; ambiguity (both `family/camping/` and `arthur/camping/`) - # generates both pages -- they're genuinely separate topics. - if single_topic: - matched = [t for t in topics if t[1] == single_topic] - if not matched: - _err(f"no topic folder found with slug '{single_topic}'") - return 1 - rc = 0 - for bucket_prefix, topic_slug in matched: - sub_rc = await _generate_topic( + # ── Selection mode ──────────────────────────────────────────────── + # Any selection flag (--home, --member, --topic, --topics) switches + # from the full sweep to "generate exactly this set". Flags combine + # and --member/--topic repeat, so the curator's incremental rebuild + # covers every page a filing burst touched in one invocation. + # Member values are normalized through `_slugify_person`, the same + # mapping the index uses, so display names and slugs both land on + # the bucket. Unknown slugs warn and skip — an auto-run fed a name + # with no bucket must not sink the batch — and the run fails only + # when NOTHING in the selection matched. + if home_sel or topics_only or members_sel or topics_sel: + generated = 0 + + if home_sel: + rc = await _generate_home( + llm, index, roster=roster, + shared_bucket=shared_bucket, lang=lang, write=not dry_run, + ) + if rc == 0: + generated += 1 + + member_slugs: list[str] = [] + for raw in members_sel: + slug = _slugify_person(raw) + if slug and slug not in member_slugs: + member_slugs.append(slug) + for slug in member_slugs: + if slug not in roster: + _err(f"no member bucket for '{slug}' — skipping") + continue + rc = await _generate_member( + llm, slug, index, vault, + shared_bucket=shared_bucket, lang=lang, write=not dry_run, + ) + if rc == 0: + generated += 1 + + # `--topic camping` matches anywhere in the vault; ambiguity + # (both `family/camping/` and `arthur/camping/`) generates both + # pages -- they're genuinely separate topics. + if topics_only: + topic_set = topics + else: + wanted = set(topics_sel) + known = {slug for _, slug in topics} + for t_slug in sorted(wanted - known): + _err(f"no topic folder found with slug '{t_slug}' — skipping") + topic_set = [t for t in topics if t[1] in wanted] + for bucket_prefix, topic_slug in topic_set: + rc = await _generate_topic( llm, bucket_prefix, topic_slug, index, shared_bucket=shared_bucket, lang=lang, write=not dry_run, ) - rc = sub_rc if sub_rc else rc - return rc + if rc == 0: + generated += 1 - if single_member: - return await _generate_member( - llm, single_member, index, vault, - shared_bucket=shared_bucket, lang=lang, write=not dry_run, - ) + if generated == 0: + _err("nothing generated — no page in the selection matched") + return 1 + return 0 # ── Default loop ────────────────────────────────────────────────── - # `--topics` skips home + members; otherwise we cover home, every - # member, and every topic in turn. - if not topics_only: - rc = await _generate_home( - llm, index, roster=roster, + # Bare invocation: home, every member, every topic in turn. + rc = await _generate_home( + llm, index, roster=roster, + shared_bucket=shared_bucket, lang=lang, write=not dry_run, + ) + if rc != 0: + return rc + + # A member with no content is skipped (not an error) so one + # empty bucket doesn't sink the whole run. `_generate_member` + # returns 1 for skipped-empty, which we deliberately swallow + # here -- the overall run still succeeds. + for slug in roster: + await _generate_member( + llm, slug, index, vault, shared_bucket=shared_bucket, lang=lang, write=not dry_run, ) - if rc != 0 or home_only: - return rc - - # A member with no content is skipped (not an error) so one - # empty bucket doesn't sink the whole run. `_generate_member` - # returns 1 for skipped-empty, which we deliberately swallow - # here -- the overall run still succeeds. - for slug in roster: - await _generate_member( - llm, slug, index, vault, - shared_bucket=shared_bucket, lang=lang, write=not dry_run, - ) for bucket_prefix, topic_slug in topics: await _generate_topic( @@ -231,7 +286,7 @@ async def _generate_home( """ prompt = _build_home_prompt(index, roster=roster, lang=lang) try: - page = (await llm.complete("overview", prompt)).strip() + page = (await llm.complete("overview", prompt, temperature=_TEMPERATURE)).strip() except LLMUnavailableError as e: _err(f"LLM unavailable: {e}") return 1 @@ -248,7 +303,7 @@ async def _generate_home( return 0 return await _publish( page, target_path="index.md", shared_bucket=shared_bucket, - commit_msg="docs(memory): refresh the family wiki home page", + commit_msg=f"{COMMIT_PREFIX} the family wiki home page", # Only used if the seed's root index.md is somehow missing; # otherwise the seed already carries this frontmatter. default_preamble="---\ntitle: Family Memory\n---", @@ -267,9 +322,39 @@ async def _generate_member( display = slug.capitalize() facts = _load_facts(vault, slug) - prompt = _build_member_prompt(display, slug, entries, facts, lang=lang) + + # Anchored regen: renumber the existing page's citations to the + # current evidence order, and name what's new since that page was + # generated. Both jobs are mechanical here so the model carries + # neither (it fails silently at both — see _previous_generated). + previous, prev_refs = _previous_generated(vault / slug / "about.md") + new_evidence: list[str] = [] + if previous: + link_to_n = { + _relative_link(e["rel"], slug): n + for n, e in enumerate(entries, start=1) + if e.get("rel") + } + remap = { + old: link_to_n[link] + for link, old in prev_refs.items() + if link in link_to_n + } + previous = _renumber_citations(previous, remap) + for n, e in enumerate(entries, start=1): + link = _relative_link(e.get("rel") or "", slug) + if link not in prev_refs: + meta = " · ".join( + x for x in (e.get("date"), e.get("title")) if x + ) + new_evidence.append(f"[{n}] {meta}") + + prompt = _build_member_prompt( + display, slug, entries, facts, lang=lang, + previous=previous, new_evidence=new_evidence, + ) try: - page = (await llm.complete("overview", prompt)).strip() + page = (await llm.complete("overview", prompt, temperature=_TEMPERATURE)).strip() except LLMUnavailableError as e: _err(f"LLM unavailable: {e}") return 1 @@ -283,7 +368,7 @@ async def _generate_member( return 0 return await _publish( page, target_path=f"{slug}/about.md", shared_bucket=shared_bucket, - commit_msg=f"docs(memory): refresh {slug}'s wiki page", + commit_msg=f"{COMMIT_PREFIX} {slug}'s wiki page", # First-creation frontmatter seeds the person entity registry on # the page itself: `canonical` is the longest synonym (usually # the formal first name when the family also uses a nickname), @@ -327,7 +412,7 @@ async def _generate_topic( display, topic_slug, scope, entries, cross_refs, lang=lang, ) try: - page = (await llm.complete("overview", prompt)).strip() + page = (await llm.complete("overview", prompt, temperature=_TEMPERATURE)).strip() except LLMUnavailableError as e: _err(f"LLM unavailable: {e}") return 1 @@ -346,7 +431,7 @@ async def _generate_topic( target_path=f"{page_dir}/about.md", shared_bucket=shared_bucket, commit_msg=( - f"docs(memory): refresh {bucket_prefix}/{topic_slug} topic page" + f"{COMMIT_PREFIX} {bucket_prefix}/{topic_slug} topic page" ), default_preamble=_topic_preamble(topic_slug, display, scope), ) @@ -701,6 +786,55 @@ def _relative_link(rel: str, page_dir: str) -> str: # ── Bracketed-region splice ──────────────────────────────────────────────── +def _previous_generated(page_path: Path) -> tuple[str, dict[str, int]]: + """The current generated body of a page, for prompt anchoring. + + Returns ``(body, refs)``: the content between the regenerate + markers with the rendered References section stripped, and the + reference map that section carried (link path → citation number). + The map lets the caller renumber the baseline's citations to the + CURRENT evidence order before prompting — new evidence shifts the + numbering, and a model asked to renumber silently doesn't + (measured 2026-06-12: byte-identical body over a shifted + references block — every citation pointing at the wrong source). + ``("", {})`` when the page doesn't exist yet. + """ + try: + text = page_path.read_text(encoding="utf-8") + except OSError: + return "", {} + start = text.find(_BEGIN) + end = text.find(_END) + if start == -1 or end == -1 or end <= start: + return "", {} + body = text[start + len(_BEGIN):end].strip() + refs_map = { + m.group(2): int(m.group(1)) + for m in re.finditer( + r"^- \[(\d+)\] \[[^\]]*\]\(([^)]+)\)", body, re.M, + ) + } + refs = body.find("## References") + if refs != -1: + body = body[:refs].rstrip() + return body, refs_map + + +def _renumber_citations(text: str, remap: dict[int, int]) -> str: + """Rewrite every ``[N]``/``[N, M]`` citation through ``remap``. + + Numbers without a mapping pass through unchanged (evidence that + vanished from the index; the prompt tells the model to drop those + lines). Single-pass over citation groups, so chained renumbering + (1→2 while 2→3) cannot double-apply. + """ + def _sub(m: re.Match) -> str: + nums = [int(x) for x in re.split(r",\s*", m.group(1))] + return "[" + ", ".join(str(remap.get(n, n)) for n in nums) + "]" + + return re.sub(r"\[(\d+(?:,\s*\d+)*)\]", _sub, text) + + def _splice_generated(existing: str, generated: str, *, default_preamble: str = "") -> str: """Place `generated` inside the regenerate region of `existing`. @@ -795,13 +929,13 @@ async def _publish(page: str, *, target_path: str, shared_bucket: str, # ── Arg helpers ────────────────────────────────────────────────────────── -def _arg_value(argv: list[str], flag: str) -> str | None: - """Value following `flag` in argv, or None if the flag is absent.""" - if flag in argv: - i = argv.index(flag) - if i + 1 < len(argv): - return argv[i + 1] - return None +def _arg_values(argv: list[str], flag: str) -> list[str]: + """Every value following an occurrence of `flag`, in argv order.""" + out: list[str] = [] + for i, arg in enumerate(argv): + if arg == flag and i + 1 < len(argv): + out.append(argv[i + 1]) + return out # ── Prompts ────────────────────────────────────────────────────────────── @@ -863,10 +997,10 @@ def _build_home_prompt(entries: list[dict], *, roster: list[str], lang: str) -> Two or three lines about the dwelling itself: ownership/rental, year acquired or moved in, anything structurally noted (mortgage, deed). Skip if nothing on record. ## Real Estate -Other properties owned (rentals, second homes, plots). One bullet per property. Write "(no information on file)" if nothing on record. +Other properties owned (rentals, second homes, plots). One bullet per property. ## Vehicles -One bullet per vehicle: year, make/model, owner. Write "(no information on file)" if nothing. +One bullet per vehicle: year, make/model, owner. ## Insurance One bullet per policy: name / provider — what it covers — owner. @@ -883,7 +1017,7 @@ def _build_home_prompt(entries: list[dict], *, roster: list[str], lang: str) -> - The H1 is the family surname prefixed with "The" (e.g. "# The Simpsons"). Pick the surname most frequently associated with household members; if multiple appear (blended family), use the dominant one. - The blockquote line below the H1 is the current primary address. Omit the blockquote entirely if no address is on file. - Respond in: {lang}. -- Use ONLY what the source summaries support. If a section has no source material, write "(no information on file)" — never invent. +- Use ONLY what the source summaries support. A section with no source material is omitted entirely — never invent, never write placeholder lines. - Cite the document that established each fact as [N] inline. Multiple sources: [1, 3]. - Keep entries dense and skimmable. One bullet per member; one bullet per item elsewhere. - Do NOT add a "References" section — it is appended programmatically after your response. Don't list source paths or URLs in your output. @@ -893,12 +1027,22 @@ def _build_home_prompt(entries: list[dict], *, roster: list[str], lang: str) -> def _build_member_prompt( display: str, slug: str, entries: list[dict], facts: list[tuple[str, str]], *, lang: str, + previous: str = "", new_evidence: list[str] | None = None, ) -> str: - """Prompt for one member's page: their slice of summaries plus facts. + """Prompt for one member's profile card: summaries plus facts. + + The page is a curated overview for assistants and browsing family, + not an archive (docs/design/brain/wiki-page-anatomy.md): recency- + weighted About, themed interests, capped activity, no ledger + content. Curated facts ride in as ground truth (the family typed + them); the document summaries are the cited evidence. Section + layout is fixed for the same reason the home page's is. - Curated facts ride in as ground truth (the family typed them); the - document summaries are the cited evidence. Section layout is fixed - for the same reason the home page's is. + ``previous`` is the page's current generated body (sans rendered + references): with it the prompt anchors the regen to the existing + wording so a new document produces a reviewable diff instead of a + full resample — temp 0 alone does NOT give that (evidence+1 changes + the input, and everything redraws; measured 2026-06-12). The evidence slice is shared by design: Lisa's birth certificate is on Lisa's page AND describes her parents. Small models conflate @@ -917,7 +1061,36 @@ def _build_member_prompt( else: facts_block = f"No hand-curated facts on file for {display}." - return f"""You are composing the personal page for one member of a family's private, self-hosted memory wiki. The page is the landing page when someone opens this person's folder in the wiki. The vault is private — identifying details are fine to include. + if previous: + if new_evidence: + fresh = "\n".join(new_evidence) + fresh_block = ( + "NEW since the baseline (not yet on the page) — work it " + "into the right sections, citing its [N]:\n" + fresh + ) + else: + fresh_block = ( + "Nothing is new since the baseline — reproduce it, " + "applying corrections only where it contradicts the " + "evidence above." + ) + anchor_block = f""" +The page already exists. Its current version is below — treat it as the baseline, not as something to rewrite: +- Keep its wording, section names, themes, and bullet order EXACTLY wherever the evidence still supports them. +- The baseline's [N] citations already match the evidence numbering above. Do not change them. +- Touch existing lines only where new evidence forces it (e.g. the Recent Activity cap pushing out the oldest entry). +- Drop a line only when its evidence no longer appears above. +- Never rephrase, restyle, or reorganize anything else. + +{fresh_block} + +CURRENT PAGE (baseline): +{previous} +""" + else: + anchor_block = "" + + return f"""You are composing the profile page for one member of a family's private, self-hosted memory wiki. It is an OVERVIEW, not an archive: the card a family member (or the family's assistant) reads to get a feel for who this person is right now — their role, their current life, their interests and quirks. Detail lives in the cited documents, one click away. This page is about: {display} (vault slug: {slug}). @@ -926,36 +1099,33 @@ def _build_member_prompt( Source summaries involving {display} (cite as [N] inline where the fact came from a specific document): {evidence} - -Produce a markdown page with this EXACT structure and section order: +{anchor_block} +Produce a markdown page with this EXACT structure and section order (omit a section entirely when nothing supports it): # {display} > ## About -One short paragraph: {display}'s own role, occupation or school, and defining details — only what the documents and facts state about {display} personally. [N] +One short paragraph: who {display} is TODAY — role in the family, occupation or school, current life context. When sources span years, prefer the most recent picture; old contracts and certificates are background, not news. [N] -## Facts & Preferences -Bullet the hand-curated facts above (rules, habits, preferences, goals), one per line. Write "(none on file)" if there are none. +## Interests & Preferences +Themed bullets that give a feel for the person: hobbies, habits, favorite places, pets, preferences, quirks. Merge the hand-curated facts above with what the documents reveal (a league membership, a regular haunt). Format: **:** . [N] ## Recent Activity -The most recent notes, bookmarks, and documents involving them, newest first. One bullet each: — . [N] +The most recent events involving them, newest first, AT MOST 8 bullets: — . [N] -## Documents -Key filed documents that name them (insurance, medical, school, financial). One bullet each. [N] - -## People & Organizations -Organizations and correspondents associated with their documents. One bullet each. [N] +## Key Documents +The load-bearing documents naming them (identity, contracts, insurance, medical, school, financial). One bullet each. [N] Receipts and everyday ephemera stay out — they remain reachable through Recent Activity citations. Rules: +- The whole page stays under roughly 300 words. It is a profile card, not a ledger: never copy line items, account numbers, or ID numbers onto the page — the citations carry that detail. - The H1 is the person's full name as it appears in the documents; fall back to "{display}". - The documents above also involve OTHER family members (see each entry's "involves:" list). A birth certificate is mostly about the parents; school papers name a parent. State a fact about {display} ONLY when the source ties it to {display} by name. Never give {display} another person's role, profession, or relationship (mother/father, spouse, caretaker, employer). - When a summary says "the mother", "her husband", or similar without naming {display}, that fact belongs to someone else — leave it off this page. - Unsure who a fact is about? Leave it out. A short page is correct; a wrong page is not. - Respond in: {lang}. -- Use ONLY what the sources and facts above support. Empty section → "(none on file)". Never invent. +- Use ONLY what the sources and facts above support. Never invent. Omit empty sections — no placeholder lines. - Cite the source document as [N] inline. Multiple sources: [1, 3]. -- Keep entries dense and skimmable. - Do NOT add a "References" section — it is appended programmatically after your response. Don't list source paths or URLs in your output. """ @@ -1024,7 +1194,7 @@ def _build_topic_prompt( f"tracking on {display.lower()}." ) - return f"""You are composing the landing page for a topic folder in a family's private, self-hosted memory wiki. The page is the first thing someone sees when they open the topic's folder. The vault is private — identifying details (full names, places, prices) are fine to include when documents reveal them. + return f"""You are composing the landing page for a topic folder in a family's private, self-hosted memory wiki. It is an OVERVIEW someone (or the family's assistant) reads to understand what this topic is about and what has happened in it lately — not an archive. Detail lives in the cited captures, one click away. The vault is private — identifying details (full names, places, prices) are fine to include when documents reveal them. This page is about the topic: {display} (vault slug: {slug}, scope: {scope}). @@ -1043,14 +1213,14 @@ def _build_topic_prompt( One short paragraph: what this topic covers in the family's memory, what kinds of captures land here, what makes it worth revisiting. [N] ## Recent Activity -The most recent captures filed under this topic, newest first. One bullet each: ` — `. [N] +The most recent captures filed under this topic, newest first, AT MOST 8 bullets: ` — `. [N] ## Cross-references {"Captures filed in other buckets that mention this topic. One bullet each: ` — [from ]`. [N]" if cross_refs else "(omit this section)"} Rules: - Respond in: {lang}. -- Use ONLY what the sources above support. Empty section → "(none on file)". Never invent. +- Use ONLY what the sources above support. Never invent. Omit empty sections — no placeholder lines. - Cite the source capture as [N] inline. Multiple sources: [1, 3]. - Keep entries dense and skimmable. - Do NOT add a "References" section — it is appended programmatically after your response. Don't list source paths or URLs in your output. diff --git a/stacklets/memory/bot/cli_entrypoint.py b/stacklets/memory/bot/cli_entrypoint.py index 15475b2..0767e9d 100644 --- a/stacklets/memory/bot/cli_entrypoint.py +++ b/stacklets/memory/bot/cli_entrypoint.py @@ -10,11 +10,13 @@ re-use the bot-runner as their tools runtime. Commands: - wiki [--home | --member ] [--dry-run] + wiki [--home] [--member ]... [--topic ]... [--dry-run] Regenerate the family wiki's entry pages. Apply by default; - `--dry-run` previews to stdout. Bare invocation regenerates the - home page plus every member's page. See `cli/wiki.py` for the - full prompt-and-splice contract. + `--dry-run` previews to stdout. Bare invocation regenerates + home + every member + every topic; selection flags repeat and + combine to cover exactly the pages a filing burst touched (the + curator's incremental path). See `cli/wiki.py` for the full + prompt-and-splice contract. """ from __future__ import annotations diff --git a/stacklets/memory/bot/curator.py b/stacklets/memory/bot/curator.py new file mode 100644 index 0000000..5e39683 --- /dev/null +++ b/stacklets/memory/bot/curator.py @@ -0,0 +1,401 @@ +"""Curator — keeps the family wiki fresh without anyone running a command. + +Runs as the `stack-memory-curator` sidecar (own slim image, see +`curator.Dockerfile`). The curator is the vault keeper: it owns the +`git pull` that keeps the working copy in sync with Forgejo (the wiki +container is a pure view that just watches the files). Jobs: + +1. **Incremental, after filings settle.** New commits land — an + archivist filing, a learn commit, a hand edit pushed from Obsidian. + The curator debounces the burst, reads which persons the changed + files involve (`persons:` frontmatter plus the bucket owner), and + regenerates exactly those member pages plus home. A typical burst + costs 2-3 LLM calls, so freshness where it's felt — "I filed + Homer's letter, Homer's page knows" — is nearly free. +2. **Nightly full sweep.** Home, every member, every topic, at + WIKI_NIGHTLY local time. This is the self-healing catch-all: topic + pages and cross-references go stale invisibly during the day and + are corrected while the GPU is idle. It also means the incremental + person mapping only has to be *helpful*, never load-bearing — + worst case for a missed page is "stale until tonight". + +Rebuilds run the CLI entrypoint as a subprocess (same code path as +`stack memory wiki`), so a wedged LLM call dies with the child, not +inside this loop. The wiki's own publish commits are filtered by +their subject prefix (`wiki.COMMIT_PREFIX`) — the rebuild must not +trigger itself. The last-rebuilt SHA persists across restarts, so an +interrupted rebuild is simply redone. + +Failure shape: a failed rebuild keeps the SHA where it was and +retries after another quiet window — the trigger is never lost, and +`stack memory wiki` always works as the manual override. If the +curator is down, the vault (and therefore the wiki) goes stale +together — one component, one failure story. Disabling +`wiki_auto_rebuild` stops the rebuilds, NOT the pull: the wiki must +not rot just because the automation is off. + +Config surface (rendered from stack.toml by the runtime): + WIKI_AUTO_REBUILD "true"/"false" — [memory] wiki_auto_rebuild + WIKI_REBUILD_QUIET_SECS quiet window — [memory] wiki_rebuild_quiet_secs + WIKI_NIGHTLY "HH:MM" local, "" disables — [memory] wiki_nightly +""" + +from __future__ import annotations + +import asyncio +import os +import subprocess +import sys +import time +from pathlib import Path + +# Sibling imports. The wiki module is loaded straight off its directory +# (not as `cli.wiki`) — the docs stacklet ships a `cli` package too, +# and the bare package name collides when both bot dirs share a +# sys.path (which the test suite's single process does). +sys.path.insert(0, str(Path(__file__).parent / "cli")) # wiki module +sys.path.insert(0, str(Path(__file__).resolve().parents[2])) # memory.lib +sys.path.insert(0, "/app") # stack.* framework + +from loguru import logger # noqa: E402 +from memory.lib import _parse_frontmatter # noqa: E402 + +from wiki import COMMIT_PREFIX # noqa: E402 + +POLL_SECS = int(os.environ.get("CURATOR_POLL_SECS", "30")) +QUIET_SECS = int(os.environ.get("WIKI_REBUILD_QUIET_SECS", "180")) +NIGHTLY = os.environ.get("WIKI_NIGHTLY", "03:30").strip() + +ENTRYPOINT = "/stacklets/memory/bot/cli_entrypoint.py" + +# Hard ceiling on one rebuild pass. A vault-wide sweep on a slow local +# model stays well under this; anything longer means a wedged LLM call +# and the retry path is cheaper than waiting forever. +REBUILD_TIMEOUT_SECS = 3600 + +# Vault-root entries that never belong to a member or the shared +# bucket. Mirrors the wiki's `_NON_MEMBER_DIRS` plus git itself. +_SKIP_TOP = {".git", ".obsidian", "wiki", "private", "templates", "_shared"} + +# Generated page filenames. Changes to these are the wiki's own output +# (or a human editing around the splice markers) — never a rebuild +# trigger, or the curator would chase its own tail. +_GENERATED_NAMES = {"about.md", "index.md"} + + +# ── Pure decision logic (tested in tests/stacklets) ────────────────────── + +def only_own_commits(subjects: list[str]) -> bool: + """True when every commit subject is a wiki publish of our own. + + An empty list is NOT ours: HEAD moved but the range shows no + commits, which means a history rewrite — the caller treats that + as a rebuild trigger, never as something to skip. + """ + return bool(subjects) and all( + s.startswith(COMMIT_PREFIX) for s in subjects + ) + + +class Debounce: + """Quiet-window tracker: `observe(head, now)` returns True once the + same head has been seen unchanged for `quiet_secs`.""" + + def __init__(self, quiet_secs: float): + self.quiet_secs = quiet_secs + self._head: str | None = None + self._since = 0.0 + + def observe(self, head: str, now: float) -> bool: + if head != self._head: + self._head, self._since = head, now + return False + return (now - self._since) >= self.quiet_secs + + def reset(self) -> None: + self._head = None + + def retry_later(self, now: float) -> None: + """Failed rebuild: keep the head, restart the quiet window.""" + self._since = now + + +def member_selection( + paths: list[str], + fm_reader, + *, + shared_bucket: str, +) -> list[str]: + """Map changed vault paths to an incremental `wiki` selection argv. + + Persons-only by design: member pages are where freshness is felt, + everything else (topics, cross-refs) heals on the nightly sweep. + Collects the bucket owner for personal captures plus every name in + `persons:` frontmatter (`fm_reader(path) -> dict`, fed by + `git show` so the mapping matches the commit). Returns e.g. + `["--home", "--member", "Homer Simpson"]`, or `[]` when nothing + relevant changed (only generated pages or skipped dirs) — home is + included whenever anything relevant changed at all. + """ + members: list[str] = [] + relevant = False + + for path in paths: + parts = [p for p in path.split("/") if p] + if not parts or parts[0] in _SKIP_TOP: + continue + if parts[-1] in _GENERATED_NAMES: + continue + relevant = True + if parts[0] != shared_bucket and len(parts) > 1 and parts[0] not in members: + members.append(parts[0]) + if path.endswith(".md"): + fm = fm_reader(path) or {} + persons = fm.get("persons") or fm.get("person") or [] + if isinstance(persons, str): + persons = [persons] + for person in persons: + if isinstance(person, str) and person.strip() and person not in members: + members.append(person) + + if not relevant: + return [] + argv = ["--home"] + for member in members: + argv += ["--member", member] + return argv + + +def nightly_due(nightly: str, last_run_date: str, now_local: time.struct_time) -> bool: + """True when the nightly sweep should run: a valid HH:MM is + configured, we're past it, and today's sweep hasn't happened.""" + if not nightly or ":" not in nightly: + return False + try: + hour, minute = (int(x) for x in nightly.split(":", 1)) + except ValueError: + return False + today = time.strftime("%Y-%m-%d", now_local) + if last_run_date == today: + return False + return (now_local.tm_hour, now_local.tm_min) >= (hour, minute) + + +# ── Git plumbing ───────────────────────────────────────────────────────── + +class Vault: + """Read-only git view of the vault working copy. + + The wiki container owns the `git pull`; we only ever read. The + bind-mounted repo belongs to the host user, so git's dubious- + ownership check is satisfied via a private GIT_CONFIG_GLOBAL + rather than touching any shared config. + """ + + def __init__(self, path: Path): + self.path = path + self._env = {**os.environ, "GIT_CONFIG_GLOBAL": "/tmp/curator-gitconfig"} + Path("/tmp/curator-gitconfig").write_text( + "[safe]\n\tdirectory = *\n", encoding="utf-8", + ) + + def _run(self, *args: str) -> str | None: + result = subprocess.run( + ["git", "-C", str(self.path), *args], + capture_output=True, text=True, env=self._env, + ) + if result.returncode != 0: + logger.debug("[curator] git {} failed: {}", args[0], result.stderr.strip()) + return None + return result.stdout + + async def head(self) -> str | None: + out = await asyncio.to_thread(self._run, "rev-parse", "HEAD") + return out.strip() if out else None + + async def sync(self) -> None: + """Fast-forward the working copy from Forgejo. + + Same cheap shape the wiki entrypoint used before the curator + took ownership: an idle tick is one `ls-remote` ref query, the + fetch + ff happens only when the remote actually moved. Never + fatal — Forgejo briefly unreachable means the tick is skipped + and everything keeps serving what is on disk. + """ + def _sync() -> None: + # Same tick shape as the old wiki entrypoint loop: skip on + # any read failure, compare refs, pull only on real change. + local = self._run("rev-parse", "HEAD") + if not local: + return + remote = self._run("ls-remote", "origin", "HEAD") + remote_head = remote.split()[0] if remote and remote.split() else "" + if not remote_head or local.strip() == remote_head: + return + self._run("pull", "--quiet", "--ff-only") + + await asyncio.to_thread(_sync) + + async def subjects(self, since: str, until: str) -> list[str]: + """Commit subjects in `since..until`. A broken range (history + rewrite) returns [] — which `only_own_commits` treats as not + ours, so the rebuild still happens.""" + out = await asyncio.to_thread( + self._run, "log", "--format=%s", f"{since}..{until}", + ) + return out.splitlines() if out else [] + + async def changed_paths(self, since: str, until: str) -> list[str]: + out = await asyncio.to_thread( + self._run, "diff", "--name-only", f"{since}..{until}", + ) + return [p for p in (out or "").splitlines() if p.strip()] + + def frontmatter_at(self, rev: str, path: str) -> dict: + """Frontmatter of `path` at `rev` ({} for deleted/binary files).""" + out = self._run("show", f"{rev}:{path}") + return _parse_frontmatter(out) if out else {} + + +# ── Rebuild ────────────────────────────────────────────────────────────── + +async def rebuild(selection: list[str]) -> bool: + """One wiki generation pass via the CLI entrypoint — the same code + path `stack memory wiki` execs, in a subprocess so a wedged LLM + call dies with the child instead of inside this loop.""" + label = " ".join(selection) if selection else "(full sweep)" + logger.info("[curator] rebuilding wiki: {}", label) + try: + proc = await asyncio.create_subprocess_exec( + sys.executable, ENTRYPOINT, "wiki", *selection, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.STDOUT, + ) + out_bytes, _ = await asyncio.wait_for( + proc.communicate(), timeout=REBUILD_TIMEOUT_SECS, + ) + except TimeoutError: + proc.kill() + logger.warning("[curator] rebuild timed out after {}s", REBUILD_TIMEOUT_SECS) + return False + except Exception as e: + logger.warning("[curator] rebuild failed to start: {}", e) + return False + + output = out_bytes.decode(errors="replace").strip() + if proc.returncode != 0: + tail = "\n".join(output.splitlines()[-5:]) + logger.warning("[curator] wiki generation rc={}: {}", proc.returncode, tail) + return False + published = sum(1 for ln in output.splitlines() if ln.startswith("published ")) + logger.info("[curator] wiki refreshed — {} page(s) published", published) + return True + + +# ── Main loop ──────────────────────────────────────────────────────────── + +async def main() -> None: + rebuild_enabled = os.environ.get("WIKI_AUTO_REBUILD", "true").lower() == "true" + + vault_dir = Path(os.environ.get("MEMORY_VAULT_DIR", "/data/memory/vault")) + shared_bucket = os.environ.get("SHARED_BUCKET", "family") + state_dir = Path(os.environ.get("CURATOR_STATE_DIR", "/data/memory/curator")) + sha_file = state_dir / "last-rebuilt-sha" + nightly_file = state_dir / "last-nightly-date" + + while not (vault_dir / ".git").exists(): + logger.info("[curator] waiting for vault at {}", vault_dir) + await asyncio.sleep(POLL_SECS) + + vault = Vault(vault_dir) + debounce = Debounce(QUIET_SECS) + + def _write(path: Path, value: str) -> str: + state_dir.mkdir(parents=True, exist_ok=True) + path.write_text(value, encoding="utf-8") + return value + + def _read(path: Path) -> str: + return path.read_text(encoding="utf-8").strip() if path.exists() else "" + + # First boot starts from the current HEAD: existing content is the + # operator's to backfill (`stack memory wiki`), not something to + # surprise a fresh install with. From then on the SHA survives + # restarts, so a rebuild interrupted mid-flight is simply redone. + last = _read(sha_file) + if not last: + head = await vault.head() + if head: + last = _write(sha_file, head) + # Same contract for the nightly: with no recorded date, any boot + # after the configured time would fire an immediate full sweep + # (found live on the test rig). First boot counts as done-today; + # the first real sweep runs tomorrow night. + if not _read(nightly_file): + _write(nightly_file, time.strftime("%Y-%m-%d", time.localtime())) + + logger.info( + "[curator] keeping {} in sync (poll {}s, quiet {}s, nightly {}, rebuild {}) from {}", + vault_dir, POLL_SECS, QUIET_SECS, NIGHTLY or "off", + "on" if rebuild_enabled else "OFF", last[:10] or "(empty repo)", + ) + + while True: + await asyncio.sleep(POLL_SECS) + # The curator owns the pull — the wiki container only watches + # files. Runs even with rebuilds disabled: the wiki must not + # rot just because the automation is off. + await vault.sync() + if not rebuild_enabled: + continue + head = await vault.head() + if not head: + continue + if not last: + last = _write(sha_file, head) + continue + + # ── Nightly full sweep ──────────────────────────────────────── + # Runs regardless of pending changes — it covers everything an + # incremental pass would, so it also clears the debounce. The + # date is recorded even on failure: one attempt per night, the + # incremental path and the manual CLI cover the gap. + if nightly_due(NIGHTLY, _read(nightly_file), time.localtime()): + _write(nightly_file, time.strftime("%Y-%m-%d", time.localtime())) + if await rebuild([]): + last = _write(sha_file, head) + debounce.reset() + continue + + # ── Incremental: persons + home, debounced ──────────────────── + if head == last: + debounce.reset() + continue + + if only_own_commits(await vault.subjects(last, head)): + last = _write(sha_file, head) + debounce.reset() + continue + + if not debounce.observe(head, time.monotonic()): + continue + + paths = await vault.changed_paths(last, head) + selection = member_selection( + paths, lambda p: vault.frontmatter_at(head, p), + shared_bucket=shared_bucket, + ) + if not selection: + last = _write(sha_file, head) + debounce.reset() + continue + + if await rebuild(selection): + last = _write(sha_file, head) + debounce.reset() + else: + debounce.retry_later(time.monotonic()) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/stacklets/memory/curator.Dockerfile b/stacklets/memory/curator.Dockerfile new file mode 100644 index 0000000..c0ee597 --- /dev/null +++ b/stacklets/memory/curator.Dockerfile @@ -0,0 +1,22 @@ +# curator.Dockerfile — slim runtime for the wiki curator sidecar. +# +# Deliberately NOT the bot-runner image: the curator needs python, +# git, loguru, and the OpenAI SDK (the framework LLM client behind +# the wiki generation it subprocesses) — 2 of the bot-runner's 10 +# dependencies. No Matrix, no libolm, no PDF stack. Code arrives by +# bind mount (see docker-compose.yml), same as the bot-runner, so +# code changes don't need an image rebuild; tzdata makes the +# WIKI_NIGHTLY local time honest inside the container. + +FROM python:3.12-slim + +RUN apt-get update && apt-get install -y --no-install-recommends \ + git tzdata \ + && rm -rf /var/lib/apt/lists/* + +RUN pip install --no-cache-dir "loguru>=0.7,<1.0" "openai>=1.50,<3.0" + +RUN adduser --disabled-password --uid 1000 curator +USER curator + +CMD ["python", "-u", "/stacklets/memory/bot/curator.py"] diff --git a/stacklets/memory/docker-compose.yml b/stacklets/memory/docker-compose.yml index 782fbb7..76ddd88 100644 --- a/stacklets/memory/docker-compose.yml +++ b/stacklets/memory/docker-compose.yml @@ -1,17 +1,17 @@ # stacklets/memory/docker-compose.yml — the family wiki # -# One service. The "wiki" is a Quartz v4 site that renders the +# Two services. The "wiki" is a Quartz v4 site that renders the # family memory vault as a browsable wiki: graph view, backlinks, # full-text search, mobile-friendly. The container runs `quartz # build --serve`, which watches the bind-mounted vault and rebuilds -# whenever the hooks pull new commits in from Forgejo. +# whenever files change. The "curator" is the vault keeper: it owns +# the `git pull` that keeps the working copy in sync with Forgejo, +# and regenerates wiki pages when filings land (see bot/curator.py). # -# The vault is mounted read-write so the wiki can keep its working -# copy in sync with Forgejo (a background `git pull` loop in the -# entrypoint). Edits still happen in Forgejo, which stays the canonical -# source of truth; the wiki only ever fast-forwards. The image is -# built locally from the sibling Dockerfile and Watchtower is told to -# ignore it (there's no upstream image to track). +# Forgejo stays the canonical source of truth; the wiki mounts the +# vault read-only and never touches git. Both images are built +# locally and Watchtower is told to ignore them (no upstream images +# to track). name: stack-memory @@ -28,11 +28,10 @@ services: - "com.centurylinklabs.watchtower.enable=false" networks: - stack - # Mounted read-write so the entrypoint can fast-forward the working - # copy from Forgejo (git pull). Edits still round-trip through - # Forgejo so they land as commits; the wiki never writes its own. + # Read-only: the wiki is a pure view. The curator below owns the + # pull; edits round-trip through Forgejo and land as commits. volumes: - - ${MEMORY_VAULT_DIR}:/vault + - ${MEMORY_VAULT_DIR}:/vault:ro environment: # Used by quartz.config.ts to build "edit on Forgejo" links and # absolute URLs. Resolved by the runtime from stack.toml. @@ -40,8 +39,6 @@ services: WIKI_HOST: ${WIKI_HOST} WIKI_IP: ${WIKI_IP} WIKI_PORT: "42070" - # How often the background loop pulls family/memory from Forgejo. - VAULT_SYNC_INTERVAL: "20" ports: # Container's Quartz preview server listens on 8080; we publish # on the stacklet's declared port (42070). PORT_BIND_IP is set @@ -50,6 +47,59 @@ services: - "${PORT_BIND_IP:-127.0.0.1}:42070:8080" restart: unless-stopped + # Curator — the vault keeper. Owns the `git pull` from Forgejo, + # runs debounced incremental rebuilds (member pages + home) after + # filings settle, plus a nightly full sweep as the self-healing + # catch-all. Own slim image (python + git + the LLM client deps) — + # see curator.Dockerfile; code arrives by bind mount like the + # bot-runner's, the env is memory's own. See `bot/curator.py`. + stack-memory-curator: + container_name: stack-memory-curator + build: + context: . + dockerfile: curator.Dockerfile + image: stack-memory-curator:local + # Locally-built image — nothing for Watchtower to pull. + labels: + - "com.centurylinklabs.watchtower.enable=false" + networks: + - stack + extra_hosts: + - "host.docker.internal:host-gateway" + environment: + # In-container paths: the vault working copy (read-only use; the + # wiki container owns the pull) and the curator's state files, + # both under the /data/memory mount below. + MEMORY_VAULT_DIR: /data/memory/vault + CURATOR_STATE_DIR: /data/memory/curator + # Forgejo, container-network address — memory's own CODE_URL in + # .env is the host-side URL the install hooks use. + CODE_URL: http://stack-code:3000 + MATRIX_ADMIN_USER: ${MATRIX_ADMIN_USER} + MATRIX_ADMIN_PASSWORD: ${MATRIX_ADMIN_PASSWORD} + # LLM endpoint (docker-side URL) + model config for the wiki + # generation the curator subprocesses. + OPENAI_URL: ${OPENAI_URL} + OPENAI_KEY: ${OPENAI_KEY} + AI_DEFAULT_MODEL: ${AI_DEFAULT_MODEL} + AI_MODELS_JSON: ${AI_MODELS_JSON} + LANGUAGE: ${LANGUAGE} + SHARED_BUCKET: ${SHARED_BUCKET} + # Behavior — rendered from stack.toml [memory]. + WIKI_AUTO_REBUILD: ${WIKI_AUTO_REBUILD:-true} + WIKI_REBUILD_QUIET_SECS: ${WIKI_REBUILD_QUIET_SECS:-180} + WIKI_NIGHTLY: ${WIKI_NIGHTLY:-03:30} + # Local time for WIKI_NIGHTLY. + TZ: ${TIMEZONE:-UTC} + volumes: + # Code mounts, same shape as the bot-runner's: curator.py + the + # wiki module from /stacklets, the framework from /app/stack. + - ../../stacklets:/stacklets:ro + - ../../lib/stack:/app/stack:ro + # Vault working copy (the curator pulls it) + state dir. + - ${MEMORY_DATA_DIR}:/data/memory + restart: unless-stopped + networks: stack: external: true diff --git a/stacklets/memory/quartz/entrypoint.sh b/stacklets/memory/quartz/entrypoint.sh index 55b8c29..1518e8c 100644 --- a/stacklets/memory/quartz/entrypoint.sh +++ b/stacklets/memory/quartz/entrypoint.sh @@ -1,43 +1,14 @@ #!/bin/sh -# entrypoint.sh — keep the vault in sync from Forgejo, then serve it. +# entrypoint.sh — serve the vault as the family wiki. # -# Quartz `--serve` watches /vault and rebuilds on change, but the working -# copy only advances when we pull. A background loop pulls family/memory -# on a short interval, so a new commit — an archivist mirror, a seed -# push, or a hand-edit in the Forgejo web UI — shows up within -# VAULT_SYNC_INTERVAL seconds and Quartz rebuilds it for free. A Forgejo -# push webhook could replace this loop later; polling keeps it -# dependency-free and is cheap (the rebuild is sub-second). +# Pure view: Quartz `--serve` watches /vault and rebuilds on change. +# The working copy is kept in sync by the curator sidecar (the vault +# keeper — it pulls from Forgejo on its poll tick; see +# `../bot/curator.py`), so this container never touches git and the +# vault is mounted read-only. If the curator is down, the wiki keeps +# serving what is on disk — stale, never broken. set -e -INTERVAL="${VAULT_SYNC_INTERVAL:-20}" - -# The working copy is cloned on the host and bind-mounted here, so git -# sees a different owner than the container user. Trust it explicitly, -# otherwise git refuses to operate on "dubious ownership". -git config --global --add safe.directory /vault - -# Background sync, kept as cheap as possible. Each tick compares the -# local HEAD against the remote with `git ls-remote` — a single -# lightweight ref query, no object transfer, no working-tree touch — and -# only pulls when the remote has actually moved. So an idle tick (the -# common case) costs one tiny request and never triggers a rebuild; the -# fetch + fast-forward happens only when there is genuinely a new commit. -# -# Never fatal: if the clone isn't ready yet (first boot) or Forgejo is -# briefly unreachable, the tick is skipped and retried next time while -# the site keeps serving what is already on disk. -( - while true; do - sleep "$INTERVAL" - local_head=$(git -C /vault rev-parse HEAD 2>/dev/null) || continue - remote_head=$(git -C /vault ls-remote origin HEAD 2>/dev/null | cut -f1) - if [ -n "$remote_head" ] && [ "$local_head" != "$remote_head" ]; then - git -C /vault pull --quiet --ff-only 2>/dev/null || true - fi - done -) & - # Quartz in the foreground; its file watcher turns each pulled change # into a rebuild. Port 8080 is the in-container port Caddy and the # compose port mapping target. diff --git a/stacklets/memory/seeds/.gitignore b/stacklets/memory/seeds/.gitignore new file mode 100644 index 0000000..df1abbb --- /dev/null +++ b/stacklets/memory/seeds/.gitignore @@ -0,0 +1,6 @@ +# Obsidian per-user state — the vault is meant to be opened in Obsidian, +# but UI state must not hit git: workspace.json rewrites itself on every +# click, and any non-curator commit triggers a wiki rebuild. +.obsidian/ +.trash/ +.DS_Store diff --git a/stacklets/memory/stacklet.toml b/stacklets/memory/stacklet.toml index 006d657..e26f7b0 100644 --- a/stacklets/memory/stacklet.toml +++ b/stacklets/memory/stacklet.toml @@ -58,6 +58,33 @@ SHARED_BUCKET = "{shared_bucket}" # also returns this path, so there is exactly one vault on disk. MEMORY_VAULT_DIR = "{data_dir}/memory/vault" +# Bind-mount source for the curator sidecar: the memory data dir as a +# whole, so one mount covers the vault (read) and the curator's state +# files (write) at the same /data/memory paths the containers expect. +MEMORY_DATA_DIR = "{data_dir}/memory" + +# Curator runtime — the slim sidecar runs the same wiki generation the +# CLI does, so it needs the LLM endpoint (docker-side), model config, +# Forgejo admin credentials (CODE_URL is overridden to the container- +# network address in docker-compose.yml), language, and timezone. +OPENAI_URL = "{ai_openai_url_docker}" +OPENAI_KEY = "{ai_openai_key}" +AI_DEFAULT_MODEL = "{ai_default_model}" +AI_MODELS_JSON = "{ai_models_json}" +LANGUAGE = "{language}" +TIMEZONE = "{timezone}" +MATRIX_ADMIN_USER = "{admin_username}" +MATRIX_ADMIN_PASSWORD = "{admin_password}" + +# Wiki freshness knobs (stack.toml [memory]). Incremental rebuilds run +# after the vault has been quiet for the debounce window, so a stack +# of documents filed in one go produces one rebuild, not one per +# document. The nightly full sweep ("HH:MM" local, empty disables) is +# the self-healing catch-all for topic pages and cross-references. +WIKI_AUTO_REBUILD = "{wiki_auto_rebuild}" +WIKI_REBUILD_QUIET_SECS = "{wiki_rebuild_quiet_secs}" +WIKI_NIGHTLY = "{wiki_nightly}" + # Public hostname the wiki renders into absolute URLs (sitemap, # social cards, etc). Empty domain (port mode) falls back to the # auto-detected LAN IP via `{ip}` so the site stays reachable. diff --git a/tests/stacklets/test_memory_curator.py b/tests/stacklets/test_memory_curator.py new file mode 100644 index 0000000..c71a06c --- /dev/null +++ b/tests/stacklets/test_memory_curator.py @@ -0,0 +1,172 @@ +"""Curator — the pure decision logic behind the wiki freshness loop. + +The sidecar's job is mostly plumbing (git reads, one subprocess call); +what's worth pinning is the logic that keeps it from misfiring: the +own-commit filter that stops a rebuild from triggering itself, the +debounce that turns a 25-document ingest into one rebuild instead of +25, the persons-only selection that keeps the incremental pass at 2-3 +LLM calls, and the nightly once-per-day gate. All pure and tested +here; the end-to-end path rides the integration rig like the rest of +the wiki. +""" + +from __future__ import annotations + +import sys +import time +from pathlib import Path + +_REPO_ROOT = Path(__file__).resolve().parent.parent.parent +sys.path.insert(0, str(_REPO_ROOT / "stacklets" / "memory" / "bot")) +sys.path.insert(0, str(_REPO_ROOT / "stacklets" / "memory" / "bot" / "cli")) + +from curator import ( # noqa: E402 + Debounce, + member_selection, + nightly_due, + only_own_commits, +) +from wiki import COMMIT_PREFIX # noqa: E402 + + +def _local(hhmm: str) -> time.struct_time: + """A struct_time for today at HH:MM local — enough for nightly_due.""" + return time.strptime(f"2026-06-11 {hhmm}", "%Y-%m-%d %H:%M") + + +# ── only_own_commits ───────────────────────────────────────────────────── + + +class TestOnlyOwnCommits: + def test_all_wiki_publishes_are_ours(self): + subjects = [ + f"{COMMIT_PREFIX} the family wiki home page", + f"{COMMIT_PREFIX} homer's wiki page", + f"{COMMIT_PREFIX} family/camping topic page", + ] + assert only_own_commits(subjects) is True + + def test_a_filing_in_the_batch_means_rebuild(self): + subjects = [ + f"{COMMIT_PREFIX} homer's wiki page", + "learn: Car Insurance Renewal 2026", + ] + assert only_own_commits(subjects) is False + + def test_empty_range_is_not_ours(self): + # HEAD moved but the log range came back empty — a history + # rewrite. Must read as "rebuild", never as "skip". + assert only_own_commits([]) is False + + def test_prefix_must_anchor_the_subject(self): + assert only_own_commits([f"revert: {COMMIT_PREFIX} home page"]) is False + + +# ── Debounce ───────────────────────────────────────────────────────────── + + +class TestDebounce: + def test_new_head_never_fires_immediately(self): + d = Debounce(quiet_secs=180) + assert d.observe("aaa", now=1000.0) is False + + def test_fires_after_quiet_window(self): + d = Debounce(quiet_secs=180) + assert d.observe("aaa", now=1000.0) is False + assert d.observe("aaa", now=1100.0) is False + assert d.observe("aaa", now=1180.0) is True + + def test_burst_restarts_the_window(self): + # Each new commit during an ingest moves HEAD — the window + # restarts and the rebuild waits for the burst to settle. + d = Debounce(quiet_secs=180) + assert d.observe("aaa", now=1000.0) is False + assert d.observe("bbb", now=1170.0) is False + assert d.observe("bbb", now=1340.0) is False + assert d.observe("bbb", now=1350.0) is True + + def test_reset_forgets_the_head(self): + d = Debounce(quiet_secs=180) + d.observe("aaa", now=1000.0) + d.reset() + assert d.observe("aaa", now=2000.0) is False + assert d.observe("aaa", now=2180.0) is True + + def test_retry_later_defers_a_failed_rebuild(self): + d = Debounce(quiet_secs=180) + d.observe("aaa", now=1000.0) + assert d.observe("aaa", now=1180.0) is True + d.retry_later(now=1200.0) + assert d.observe("aaa", now=1300.0) is False + assert d.observe("aaa", now=1380.0) is True + + +# ── member_selection ───────────────────────────────────────────────────── + + +def _fm(mapping): + """fm_reader stub: path -> frontmatter dict.""" + return lambda path: mapping.get(path, {}) + + +class TestMemberSelection: + def test_document_with_persons_selects_their_pages(self): + paths = ["family/documents/2026/06/2026-06-10-car-insurance-p7.md"] + reader = _fm({paths[0]: {"persons": ["Homer Simpson", "Marge Simpson"]}}) + assert member_selection(paths, reader, shared_bucket="family") == [ + "--home", + "--member", "Homer Simpson", + "--member", "Marge Simpson", + ] + + def test_personal_capture_selects_the_bucket_owner(self): + paths = ["homer/notes/2026/06/duff-recipe-ab12.md"] + assert member_selection(paths, _fm({}), shared_bucket="family") == [ + "--home", "--member", "homer", + ] + + def test_generated_pages_never_trigger(self): + # The wiki's own output (or a hand edit around the splice + # markers) must not feed back into a rebuild. + paths = ["index.md", "homer/about.md", "family/camping/about.md"] + assert member_selection(paths, _fm({}), shared_bucket="family") == [] + + def test_skipped_dirs_never_trigger(self): + paths = ["wiki/config.json", ".obsidian/workspace.json", "private/x.md"] + assert member_selection(paths, _fm({}), shared_bucket="family") == [] + + def test_relevant_change_without_persons_still_refreshes_home(self): + # A shared-bucket document with no persons frontmatter: home + # aggregates it, member pages are untouched. + paths = ["family/documents/2026/06/2026-06-10-electricity-p9.md"] + assert member_selection(paths, _fm({}), shared_bucket="family") == ["--home"] + + def test_no_duplicate_members(self): + paths = [ + "homer/notes/2026/06/a-1111.md", + "homer/notes/2026/06/b-2222.md", + ] + reader = _fm({p: {"persons": ["Homer Simpson"]} for p in paths}) + sel = member_selection(paths, reader, shared_bucket="family") + assert sel.count("--member") == 2 # homer (bucket) + Homer Simpson (fm) + assert sel[0] == "--home" + + +# ── nightly_due ────────────────────────────────────────────────────────── + + +class TestNightlyDue: + def test_due_after_configured_time(self): + assert nightly_due("03:30", "", _local("03:31")) is True + + def test_not_due_before_configured_time(self): + assert nightly_due("03:30", "", _local("02:59")) is False + + def test_runs_once_per_day(self): + assert nightly_due("03:30", "2026-06-11", _local("04:00")) is False + assert nightly_due("03:30", "2026-06-10", _local("04:00")) is True + + def test_empty_or_garbage_disables(self): + assert nightly_due("", "", _local("12:00")) is False + assert nightly_due("never", "", _local("12:00")) is False + assert nightly_due("3x:99y", "", _local("12:00")) is False diff --git a/tests/stacklets/test_memory_wiki.py b/tests/stacklets/test_memory_wiki.py index cd7a2af..ef6901c 100644 --- a/tests/stacklets/test_memory_wiki.py +++ b/tests/stacklets/test_memory_wiki.py @@ -319,3 +319,55 @@ def test_display_with_special_chars(self): pre = _topic_preamble("van-life", "Van Life", "shared") assert "title: Van Life" in pre assert "slug: van-life" in pre + + +# ── Anchored regen helpers ────────────────────────────────────────────── + +from wiki import _previous_generated, _renumber_citations # noqa: E402 + + +class TestPreviousGenerated: + def _page(self, tmp_path: Path, body: str) -> Path: + p = tmp_path / "about.md" + p.write_text( + "---\ntitle: Homer\n---\n\n" + "\n\n" + f"{body}\n\n" + "\n", + encoding="utf-8", + ) + return p + + def test_missing_page_is_empty(self, tmp_path: Path) -> None: + assert _previous_generated(tmp_path / "nope.md") == ("", {}) + + def test_page_without_markers_is_empty(self, tmp_path: Path) -> None: + p = tmp_path / "about.md" + p.write_text("# Hand-written page\n", encoding="utf-8") + assert _previous_generated(p) == ("", {}) + + def test_strips_references_and_returns_map(self, tmp_path: Path) -> None: + p = self._page( + tmp_path, + "# Homer\n\nFact [1]. Other [2].\n\n" + "## References\n\n" + "- [1] [Birth Cert](../family/documents/bc.md) - 1956-05-15\n" + "- [2] [Note](notes/n.md) - 2026-06-12", + ) + body, refs = _previous_generated(p) + assert "## References" not in body + assert body.startswith("# Homer") + assert refs == {"../family/documents/bc.md": 1, "notes/n.md": 2} + + +class TestRenumberCitations: + def test_single_and_grouped(self) -> None: + out = _renumber_citations("A [1]. B [1, 3]. C [2].", {1: 2, 2: 3, 3: 1}) + assert out == "A [2]. B [2, 1]. C [3]." + + def test_chained_remap_applies_once(self) -> None: + # 1→2 while 2→3: [1] must become [2], not slide on to [3]. + assert _renumber_citations("[1] [2]", {1: 2, 2: 3}) == "[2] [3]" + + def test_unmapped_numbers_pass_through(self) -> None: + assert _renumber_citations("Kept [7].", {1: 2}) == "Kept [7]."