From 3d7cc626c396acf52652e8e0fc7006d0de1a6310 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sat, 20 Jun 2026 20:33:15 +0200 Subject: [PATCH 01/45] docs: name the event pipeline (ADR-010) + email tools design ADR-010 names the recurring architecture - sources -> ledger -> projections - event-sourcing-flavored but not full ES. Key guarantee: the machine-derived vault is reproducible from its sources, indexed by the Matrix ledger. Reprocessing replays the source, never the vault file - and for a chat filing the source is the whole thread: the original message plus its reply chain (corrections), folded in order. Only user hand-edits are irreducible. dev.md carries the short rule; framework deferred until a third source earns it. The email-tools design doc is that third source. --- docs/adr/adr-010-event-pipeline.md | 97 +++++++++++ docs/agent/dev.md | 4 + docs/design/agent/email-tools.md | 269 +++++++++++++++++++++++++++++ 3 files changed, 370 insertions(+) create mode 100644 docs/adr/adr-010-event-pipeline.md create mode 100644 docs/design/agent/email-tools.md diff --git a/docs/adr/adr-010-event-pipeline.md b/docs/adr/adr-010-event-pipeline.md new file mode 100644 index 0000000..84303af --- /dev/null +++ b/docs/adr/adr-010-event-pipeline.md @@ -0,0 +1,97 @@ +# ADR-010: Event Pipeline + +## Status +Accepted + +## Context +famstack keeps growing inputs that follow the same shape: a document arrives in +Paperless, a URL or note is pasted in Matrix, a voice memo is recorded, and +(next) email is fetched over IMAP. Each one is captured, surfaced in a Matrix +room, filed into the vault, and later consumed — by the wiki deriver, the +search/Q&A loop, reminders, and the planned Family Agent. + +Looking at this from a birds-eye view, four boxes recur: + +1. **Sources** — an inbound channel produces a `SourceContent` (text + title + + origin URI). Documents, captures, soon email. +2. **Ledger** — every state-changing filing emits a `dev.famstack.event` + envelope on the Matrix timeline (`build_document_event` / `build_capture_event`). +3. **Surface / routing** — rooms bind to intent via `dev.famstack.capture` + room state carrying a `kind` (topic, documents, soon mailbox); filings post + to the bound room. +4. **Projections / processors** — consumers fold the stream into derived state + and actions: the wiki deriver, the tasks rollup, the agent, reminders. + +This looks like event sourcing, which raised the question of whether to (a) +adopt event sourcing as a formal architecture and (b) build a generic +Channel/Processor framework now that we have nearly three sources. + +The risk on both: the repo's own guidance (`agent/plan.md`, `dev.md`) warns +that generalising across consumers before a second/third concrete caller forces +the abstraction is the canonical premature-framework failure mode. The +archivist's tool loop was deliberately left un-generalised for exactly this +reason. + +## Decision +Name the architecture **the event pipeline** and treat it as a *content +pipeline with an audit ledger and derived projections* — event-sourcing- +flavored, but not textbook event sourcing. + +- **The vault (Forgejo git) is the working source of truth** for reads and + hand-edits; its git history is the durable record. The `dev.famstack.event` + stream is the audit/notify ledger, not the system of record. We do not adopt + a formal event store or CQRS — that ceremony does not pay off for a + single-Mac, single-family system. +- **The machine-derived vault is reproducible from its sources, indexed by the + ledger.** Everything the machine produced — document mirrors, capture + entries, derived wiki/topic/tasks pages — can be rebuilt by replaying filings + against their sources. The ledger says *what* was filed and *where its source + is*; the source holds the content. **Reprocessing replays the source, never + the vault file** — and the source of a chat-originated filing is the *whole + thread*: a URL/note capture reprocesses **its originating Matrix message plus + the reply chain rooted on it** (the corrections), folded in timeline order; a + document re-fetches Paperless by `paperless_id`, an email re-fetches IMAP by + `message-id`. The vault entry is the fold of original + corrections + (`_collect_correction_chain` already walks this thread for live corrections; + rebuild reuses the same walk). The only content not reproducible this way is + **user hand-edits** (plus `facts.toml`, ontology evolution) — preserved in + git, the irreducible human delta. This is "from sources, indexed by the + ledger" — *not* "from Matrix alone"; events stay thin, and full-content + disaster recovery is the backup stacklet's job, not the ledger's. +- **Extract only the thin seams that already repeat**, nothing more: + 1. the `SourceContent` ingestion contract, + 2. the `dev.famstack.event` envelope, + 3. the `dev.famstack.capture` room-binding-by-`kind`. + A new source should need only: an extractor that yields `SourceContent`, a + `kind` for routing, and (if it acts outward) a tool. +- **Defer the framework.** Do not build a generic Channel/Processor abstraction + layer yet. The next concrete source (email — see + [../design/agent/email-tools.md](../design/agent/email-tools.md)) is the + third instance; build it against the three seams above and let it prove or + strain them. Formalise the abstraction only if a real second consumer or a + second decision branch forces it — on evidence, not aesthetics. + +## Consequences +- There is one named mental model for how inputs flow through famstack; new + features describe themselves as "a source" / "a processor" / "a binding". +- Adding a source is cheap and uniform: extractor → `SourceContent` → classify + → vault + event, plus a `kind` for room routing. +- We keep the useful parts of event sourcing (append-only ledger, replayable + deriver, idempotent keyed writes) without its operational cost. +- The framework is intentionally absent. Until a third instance earns it, some + per-source divergence (Paperless write-back, sender routing, email account + binding + outbound) stays as concrete code rather than forced into a shared + interface — accepted as the cheaper risk than a premature abstraction. +- **Reprocessing is defined as replaying the source, not patching the vault.** + This closes a current gap: capture reprocess reads the vault file today; the + reproducibility guarantee requires it to re-derive from the originating Matrix + message instead. Documents (Paperless re-fetch) and email (IMAP re-fetch) + already work this way. +- The reproducibility guarantee is kept honest by **one rebuild test**: + fabricated sources → ingest → vault; wipe the machine-derived files; rebuild; + assert equality modulo timestamps. Every new source must pass it — if you + can't replay it from its source, you're not done. +- We deliberately stop short of *full* event sourcing (rebuild-from-log alone, + time-travel, event-store infra). If those are ever needed, this ADR is the + place that records we scoped reproducibility to "from sources + ledger" on + purpose, and the decision would be revisited explicitly. diff --git a/docs/agent/dev.md b/docs/agent/dev.md index 03034e9..f1031fc 100644 --- a/docs/agent/dev.md +++ b/docs/agent/dev.md @@ -57,6 +57,10 @@ famstack/ 4. **Convention over configuration.** If a file exists with the documented name (`hooks/on_install.py`, `cli/foo.py`, `caddy.snippet`, `bot/bot.toml`), it is picked up. No registration step. 5. **The `stack` CLI is the sanctioned agent interface.** Bots, hooks, and external automations call `./stack ` - they do not import from `lib/`. Commands have stable exit codes, JSON output, and idempotent semantics. +## Reprocessing replays the source + +Reprocessing re-derives from the source, never from the vault file. For a chat filing the source is the **whole thread** - the original message **plus its reply chain** (corrections) - folded in timeline order. Machine-derived vault state must stay reproducible this way; only user hand-edits are irreducible. See [adr-010](../adr/adr-010-event-pipeline.md). + ## Stacklet anatomy Required: diff --git a/docs/design/agent/email-tools.md b/docs/design/agent/email-tools.md new file mode 100644 index 0000000..9d22794 --- /dev/null +++ b/docs/design/agent/email-tools.md @@ -0,0 +1,269 @@ +# Family Email Tools — design + +> Status: design draft +> Created: 2026-06-20 +> Target: ingestion ships independent of the agent; drafting rides on the +> Family Agent runtime (agent v0.4, see [plan.md](plan.md)) +> Depends on: +> - [plan.md](plan.md) — the agent runtime + tool surface that drafting needs +> - [../brain/family-memory.md](../brain/family-memory.md) — the vault shape email files into +> - [../brain/knowledge-architecture.md](../brain/knowledge-architecture.md) — the event bus filings ride + +## Goal + +Bring family email into the Family Brain and let famstack do real work with it: + +1. **Ingest** family email into the vault — searchable, classified, alongside + documents and captures. +2. **Surface email in Matrix** — family email shows up in a Matrix room, with + the room ↔ mailbox binding configured the famstack-native way (room state, + set by the bot on invite). +3. **Derive tasks** from email (deadlines, forms to return, payments due). +4. **Draft replies** the agent composes from vault context; a human approves + before anything is sent. + +The constraint that shapes the whole design: **wire established components, +write as little new code as possible.** Two of the three layers above already +exist in the shipped brain; only drafting is genuinely new. + +## The reframe: three layers, two already exist + +Email is not a new subsystem. It is: + +1. **Another ingestion source.** An email is structurally a document/capture: + a body, a subject, a sender, a date. The archivist's classify→mirror + pipeline already turns a `SourceContent` into a vault entry. Email points at + the pipeline we already ship — it does not get its own brain. +2. **Tasks for free.** The classifier already extracts `action_items` + ("deadlines, payments due, forms to return" — `pipeline.py` prompt) and + renders them in the `> [!summary]` briefing. An email that says "send the + form back by Friday" *already* becomes an action item. The only new bit is + rolling those up into a tasks view — the same single-walk rollup the wiki + and grocery list use. +3. **Drafting — the new capability.** Composing a reply is a tool call by the + **Family Agent** (the layer designed in [plan.md](plan.md), not yet built). + Drafting is low-risk; *sending* is the one privileged outbound action, and + it is human-in-the-loop. + +This is also the pattern `IDEAS.md` keeps arriving at independently (the +Mediathek note's "build a generic `cast_to_tv` tool, one tool many callers"). +Email drafting is just another caller of the agent's tool surface. + +## Wire, don't build: himalaya as the email surface + +Don't write IMAP/SMTP/MIME. Wire **himalaya** (`github.com/pimalaya/himalaya`), +a mature CLI email client. Verified against its docs: + +- Rust binary, **Homebrew-installable** (matches our Apple-Silicon install story). +- Backends: **IMAP, SMTP**, JMAP, Maildir. +- **`--json` output** on every command — machine-readable envelopes (id, + message-id, flags, subject, from, to, date, attachments). +- Commands we need: `envelope list` / `envelope search`, `message read ` + (raw RFC 5322 or JSON), `message add -m drafts` (save a draft), `message send`. +- OAuth2 via an external token broker; TOML multi-account config. + +himalaya becomes the implementation behind a `stack mail` CLI. This preserves +the framework invariant **"the `stack` CLI is the sanctioned agent interface"**: +the agent calls `stack mail …` as a tool, exactly like every other capability. +We own the glue (config rendering, the email→`SourceContent` adapter, the +approve loop); himalaya owns the protocol. + +### Email → the existing ingestion contract + +`SourceContent` (verified in `extractors.py`) is `text` + `title_hint` + +`source_uri`. Email maps cleanly: + +| `SourceContent` | from email | +|---|---| +| `text` | message body (`himalaya message read --json` → plaintext/markdown) | +| `title_hint` | subject | +| `source_uri` | `message-id` (stable, dedupe key) | + +Feed that into the existing `CapturePipeline` path (it already has +`capture_url` / `capture_text` / `capture_binary`; email is one more +`capture_*` sibling) and an email becomes a vault entry with a `> [!summary]` +briefing and `## Action items` — no new intelligence written. + +## Architecture + +``` + IMAP mailbox + │ himalaya (Maildir sync) + ▼ + `stack mail sync` (host cron, like the memory curator) + │ new messages → SourceContent(text=body, title_hint=subject, source_uri=message-id) + ▼ + CapturePipeline (EXISTING) → classify → mirror to vault → dev.famstack.event + │ + ├─ vault entry: /mail/YYYY/MM/-.md (type: email) + │ with `## Action items` already extracted + ├─ tasks rollup (deriver/wiki pattern) → /tasks.md + └─ mail bot routes the filing event → the Matrix room whose + dev.famstack.capture {kind: mailbox, account, folder} binding matches + (sender + subject + briefing + vault link) + + Family Agent runtime (agent v0.4, restricted container) + ├─ reads: vault (incl. the email + related facts) :ro + ├─ tool: stack mail draft / stack mail send + └─ LLM: local (oMLX), no cloud + │ composes a reply → saves a draft → posts the draft into Matrix + ▼ + Human reviews in Matrix → approves → `stack mail send ` (SMTP) +``` + +## Routing email into Matrix rooms + +Family email should *appear in Matrix* — in the room the family chose for it — +not just file silently into the vault. famstack already has the exact +mechanism: topic rooms bind a room to intent via a **`dev.famstack.capture`** +room-state event carrying a `kind`, and the bot welcomes itself + bootstraps on +join (`topic_rooms.make_room_state`, `archivist._send_room_welcome_if_needed`). +The state schema already anticipates kinds beyond `topic` (the documents-room +binding, "future" kinds). Email is one more kind on the same rails. + +**The binding is room state, set by the bot — not static `stack.toml`.** This +matches the topic-rooms pattern and the self-explaining-UX rule (bots configure +on invite, not via a config file the family never sees). Split of +responsibilities: + +- **Credentials** (per-account IMAP/SMTP) live in config + the secret store — + himalaya's multi-account TOML, rendered from `stack.toml` + secrets. This is + machine config, not room state. +- **The room ↔ (account, folder) binding** lives in `dev.famstack.capture` + room state with `kind: "mailbox"`, `{account, folder}`. Set by the mail bot + when invited to a room. + +### Bind-on-invite flow (mirrors topic rooms) + +1. A family member creates a room (e.g. `#Post` or `#Schule`) and invites the + mail bot. +2. The bot joins, **introduces itself** (existing welcome path), and checks for + a `dev.famstack.capture` mailbox binding. +3. If unbound, it asks the one question it needs: *which account and folder + route here?* (e.g. `family@… / INBOX`, or `family@… / Schule`). Answered by + a short reply or a `bind ` command. +4. The bot writes the room-state binding. From then on, mail in that + (account, folder) posts to that room. + +One account's folders can fan out to different rooms (INBOX → `#Post`, a +"Schule" label → `#Schule`); the binding is per-room, so it composes. + +### How a message reaches the room + +`stack mail sync` (host cron) fetches IMAP → Maildir and files to the vault as +above, emitting a `dev.famstack.event` whose `data` carries `account` + `folder`. +The mail bot — which lives in bot-runner and *can* read room state — routes that +event to the room whose `dev.famstack.capture` binding matches, posting the +sender + subject + the `> [!summary]` briefing + a link to the vault entry. +Same path documents and captures already take to their rooms; the binding just +says *which* room. Because the room is bound, a reply in it ("draft an answer", +"remind me Friday") is scoped to that mailbox — the topic-rooms reply-chain +pattern, reused. + +## What we reuse vs. build + +**Reuse (no new code):** +- The classify pipeline + `action_items` extraction. +- `vault_entry` / `mirror_format` rendering and the OKF-conformant frontmatter. +- The `dev.famstack.event` bus (`build_capture_event`). +- The single-walk rollup pattern (wiki/grocery) for the tasks view. +- The **`dev.famstack.capture` room-state binding** + the bot's welcome / + bind-on-invite path (`make_room_state`, `_send_room_welcome_if_needed`) — add + a `kind: mailbox`, don't invent a new config surface. +- `stacklets/core/tools-server/` as the host for the agent's `mail` tool. +- himalaya for *all* email I/O. + +**Build (small, additive):** +- `stacklets/mail/` — wraps himalaya as `stack mail` (sync/list/read/draft/send), + renders himalaya's TOML config from `stack.toml` + secrets. +- A thin email→`SourceContent` adapter + a `capture_email` entry on the pipeline. +- A **mail bot** (MicroBot, same pattern as the archivist — *not* the agent + runtime): binds rooms on invite (`kind: mailbox` room state) and routes + filing events to the bound room. +- A tasks rollup page (needs `_index_vault` to also surface `action_items` — it + currently captures summary/persons/topics/tags but not action items; small + extension). +- The `draft_email` / `send_email` agent tool — on the chosen runtime, once the + agent Phase 0 spike picks one. + +## The `stack mail` contract (agent-facing) + +Stable exit codes, `--json` output, idempotent — the same contract every +agent-callable command honors. + +| Command | Does | Outbound? | +|---|---|---| +| `stack mail sync` | fetch new IMAP mail to Maildir, hand new messages to the classify pipeline; dedupe by `message-id` | no (read) | +| `stack mail list [--json] [--query …]` | envelope list / search | no | +| `stack mail read [--json]` | message body | no | +| `stack mail draft --to … --subject … --in-reply-to --body-file …` | save a draft (himalaya `message add -m drafts`); never sends | no | +| `stack mail send ` | send a saved draft over SMTP | **YES — gated** | + +## Drafting: the safe loop (no autonomous send) + +1. Agent composes a reply from vault context (the thread + cited facts). +2. `stack mail draft` saves it to the Drafts folder. +3. Agent posts the draft text into Matrix for review, with citations. +4. A human edits/approves. +5. `stack mail send ` sends it. + +No autonomous outbound. This matches [plan.md](plan.md)'s posture (restricted +container, local LLM, no cloud calls) and German trust norms (a machine sending +mail unsupervised is a non-starter). SMTP credentials live in the secret store / +Keychain, not in the agent container; only the host-side `stack mail send` can +read them. + +## Sequencing + +- **Phase A — ingestion + Matrix surfacing (no agent runtime needed).** + `stacklets/mail/` + `stack mail sync/list/read` + the email→pipeline adapter + + the mail bot (bind-on-invite room state + route filings to the bound room) + + tasks rollup. Uses the *existing* MicroBot/room-state machinery, not the agent + runtime. Ships standalone value: family email appears in the right Matrix + room, is searchable in the brain, and its tasks are extracted. This is the + natural next move and sets the agent up with email context to reason over. +- **Phase B — agent runtime.** The [plan.md](plan.md) Phase 0 spike (nanobot + vs pi), run against the **email-draft flow** as the concrete test instead of + generic Q&A. +- **Phase C — drafting.** `draft_email` / `send_email` tools on the chosen + runtime, plus the approve loop above. + +Phase A is independent and immediately useful; Phase C is the "agent does real +work" milestone and depends on the runtime. + +## Invariants / safety + +- **All email protocol work goes through himalaya.** No handwritten IMAP/SMTP/MIME. +- **Local LLM** for classification and drafting. No cloud. +- **Ingestion is read-only on the mailbox by default.** Track ingested + `message-id`s in a cache (like the mirror cache) rather than mutating server + flags, unless the user opts into mark-as-seen. +- **Sending is the only privileged action** — human-approved, credentials + isolated to the host, never in the agent container. +- **Never file into the vault without classification** — consistent with + documents and captures; an email gets the same briefing + privacy bucket. +- **Idempotent sync** — re-running `stack mail sync` never double-files + (message-id dedupe), same discipline as document reprocess. + +## Open questions + +1. **Account model.** One shared family mailbox, or per-person accounts? + Per-person routes naturally to per-person buckets (like captures by sender). +2. **What to ingest.** All mail floods the vault with newsletters/receipts. A + himalaya `envelope search` filter, an allowlist, or a cheap "is this worth + filing?" classifier gate? Lean toward a gate so the vault stays signal. +3. **Tasks surface.** A dedicated `tasks.md` rollup, per-person task lists, or a + Matrix `what's on my plate?` query answered live from `action_items`? +4. **Drafting context budget.** How much vault context feeds the draft LLM? + Start minimal — the thread plus explicitly cited facts. +5. **Stacklet boundary.** New `mail` stacklet (distinct domain + credentials, + emitting into the shared vault/event bus) vs. a source inside `docs`. Lean + new stacklet. +6. **Bidirectional rooms?** A bound room shows incoming mail; should *posting* + in it (or a reply-chain) be able to *send* mail too (via the human-approved + `stack mail send`)? Natural, but it couples Phase A's bot to Phase C's + drafting — keep read-only in Phase A, revisit once drafting exists. +7. **Binding UX.** Free-text question-and-answer on invite, a `bind + ` command, or a room-name convention like topic rooms' `Thema:` + prefix (e.g. `Post:`)? Account+folder is hard to encode in a name, so lean + on the bot setting state from a short reply. From e02c25ead42635e1b534ccfe2481e19fd2f867af Mon Sep 17 00:00:00 2001 From: Arthur Date: Sat, 20 Jun 2026 21:00:15 +0200 Subject: [PATCH 02/45] docs: email runs as a container, not a host service himalaya needs no host resource and is the credentialed, network-facing piece - so it runs in a small mail container (egress scoped, creds from secrets), and the classify/file/route logic reuses the bot-runner pipeline via a mail bot (shared Maildir handoff). Records the verified himalaya envelope-list JSON contract and flags the message-read body schema as pin-on-install. --- docs/design/agent/email-tools.md | 93 +++++++++++++++++++++++--------- 1 file changed, 68 insertions(+), 25 deletions(-) diff --git a/docs/design/agent/email-tools.md b/docs/design/agent/email-tools.md index 9d22794..b809b68 100644 --- a/docs/design/agent/email-tools.md +++ b/docs/design/agent/email-tools.md @@ -84,14 +84,47 @@ Feed that into the existing `CapturePipeline` path (it already has `capture_*` sibling) and an email becomes a vault entry with a `> [!summary]` briefing and `## Action items` — no new intelligence written. +### himalaya JSON contract + +himalaya emits JSON with `--json`. **`envelope list --json`** is verified +against the himalaya README: + +```json +{"envelopes": [ + {"id": "...", "message-id": "...", "flags": [{"raw": "...", "iana": "..."}], + "subject": "...", "from": [{"name": "...", "email": "..."}], + "to": [...], "date": "...", "size": 0, "has-attachment": false}]} +``` + +Note `from`/`to` are **arrays of `{name, email}`**, not strings; field is +`message-id` (hyphen). **`message read --json`** (the body) is **not +documented in the README** — its exact shape must be pinned by running himalaya +once installed (Phase A2), not assumed. The code keeps the himalaya-JSON parser +isolated from the pure `email → SourceContent` mapper for exactly this reason: +the mapper's contract is famstack's own dataclass and is fully testable now; +the himalaya body schema is the one thing we confirm against the live binary. + ## Architecture +**Runtime: a container, not a host service.** himalaya needs no host resource +(unlike `ai`'s Metal or `memory`'s git), and it is the one piece that holds +credentials and talks to *external* mail servers — exactly what the agent +plan's "restriction via container" posture is for. So the network-facing +himalaya runs in a small **`mail` container** with egress scoped to the mail +host and creds from the secret store. The classify/mirror/route logic is *not* +re-built: it reuses the existing pipeline in **bot-runner** via a mail bot +beside the archivist, with the Maildir as the handoff (a shared volume) — no +cross-container API, no duplicated pipeline. `stack mail …` wraps `docker exec` +into the container, like other stacklets' CLIs. + ``` - IMAP mailbox - │ himalaya (Maildir sync) + IMAP / SMTP mailbox + │ himalaya — mail CONTAINER (only network-facing piece; + │ egress scoped to the mail host, creds from secrets) ▼ - `stack mail sync` (host cron, like the memory curator) - │ new messages → SourceContent(text=body, title_hint=subject, source_uri=message-id) + Maildir (shared volume) + │ mail bot (in bot-runner, beside the archivist) reads new messages, + │ maps each → SourceContent(text=body, title_hint=subject, source_uri=message-id) ▼ CapturePipeline (EXISTING) → classify → mirror to vault → dev.famstack.event │ @@ -150,8 +183,9 @@ One account's folders can fan out to different rooms (INBOX → `#Post`, a ### How a message reaches the room -`stack mail sync` (host cron) fetches IMAP → Maildir and files to the vault as -above, emitting a `dev.famstack.event` whose `data` carries `account` + `folder`. +`stack mail sync` (the mail container's himalaya) fetches IMAP → Maildir; the +mail bot files new messages to the vault as above, emitting a +`dev.famstack.event` whose `data` carries `account` + `folder`. The mail bot — which lives in bot-runner and *can* read room state — routes that event to the room whose `dev.famstack.capture` binding matches, posting the sender + subject + the `> [!summary]` briefing + a link to the vault entry. @@ -174,12 +208,15 @@ pattern, reused. - himalaya for *all* email I/O. **Build (small, additive):** -- `stacklets/mail/` — wraps himalaya as `stack mail` (sync/list/read/draft/send), - renders himalaya's TOML config from `stack.toml` + secrets. -- A thin email→`SourceContent` adapter + a `capture_email` entry on the pipeline. -- A **mail bot** (MicroBot, same pattern as the archivist — *not* the agent - runtime): binds rooms on invite (`kind: mailbox` room state) and routes - filing events to the bound room. +- A thin **email→`SourceContent` mapper** + a `capture_email` entry on the + pipeline. Pure, fully unit-testable now — *slice A1*. +- A small **`mail` container** running himalaya (egress scoped to the mail host, + creds from secrets, Maildir on a shared volume), plus `stack mail` wrapping + `docker exec` (sync/list/read/draft/send) and rendering himalaya's TOML config + from `stack.toml` + secrets — *slice A2, needs the binary + a test account*. +- A **mail bot** in bot-runner (MicroBot, beside the archivist — *not* the agent + runtime): reads new Maildir messages, runs `capture_email`, binds rooms on + invite (`kind: mailbox` room state) and routes filings to the bound room. - A tasks rollup page (needs `_index_vault` to also surface `action_items` — it currently captures summary/persons/topics/tags but not action items; small extension). @@ -209,19 +246,23 @@ agent-callable command honors. No autonomous outbound. This matches [plan.md](plan.md)'s posture (restricted container, local LLM, no cloud calls) and German trust norms (a machine sending -mail unsupervised is a non-starter). SMTP credentials live in the secret store / -Keychain, not in the agent container; only the host-side `stack mail send` can -read them. +mail unsupervised is a non-starter). SMTP credentials live in the secret store +and are injected only into the `mail` container's env; nothing else can read +them, and the container's egress is scoped to the mail host. ## Sequencing -- **Phase A — ingestion + Matrix surfacing (no agent runtime needed).** - `stacklets/mail/` + `stack mail sync/list/read` + the email→pipeline adapter + - the mail bot (bind-on-invite room state + route filings to the bound room) + - tasks rollup. Uses the *existing* MicroBot/room-state machinery, not the agent - runtime. Ships standalone value: family email appears in the right Matrix - room, is searchable in the brain, and its tasks are extracted. This is the - natural next move and sets the agent up with email context to reason over. +- **Phase A1 — the pure core (no binary, no mailbox, no rig).** The + email→`SourceContent` mapper + `capture_email` on the pipeline, fully + unit-tested with fabricated inputs. Proves the ADR seam (does email slot into + `SourceContent` cleanly?) and is mergeable on its own. +- **Phase A2 — ingestion + Matrix surfacing.** The `mail` container (himalaya) + + `stack mail sync/list/read` + the mail bot (reads Maildir → `capture_email`, + bind-on-invite room state, routes filings to the bound room) + tasks rollup. + Needs himalaya installed + a test account; pins the `message read --json` + body schema against the live binary. Uses the *existing* MicroBot/room-state + machinery, not the agent runtime. Ships standalone value: family email appears + in the right Matrix room, searchable, with tasks extracted. - **Phase B — agent runtime.** The [plan.md](plan.md) Phase 0 spike (nanobot vs pi), run against the **email-draft flow** as the concrete test instead of generic Q&A. @@ -256,9 +297,11 @@ work" milestone and depends on the runtime. Matrix `what's on my plate?` query answered live from `action_items`? 4. **Drafting context budget.** How much vault context feeds the draft LLM? Start minimal — the thread plus explicitly cited facts. -5. **Stacklet boundary.** New `mail` stacklet (distinct domain + credentials, - emitting into the shared vault/event bus) vs. a source inside `docs`. Lean - new stacklet. +5. **Stacklet boundary.** RESOLVED. A `mail` container owns the network-facing + himalaya (creds, scoped egress); the classify/file/route logic reuses the + docs pipeline via a mail bot in bot-runner (shared Maildir handoff). Not a + host stacklet (no host-resource need; isolation wanted), not a fork of the + docs pipeline (reused). 6. **Bidirectional rooms?** A bound room shows incoming mail; should *posting* in it (or a reply-chain) be able to *send* mail too (via the human-approved `stack mail send`)? Natural, but it couples Phase A's bot to Phase C's From fbaf1998df39a4a7bf4318f94ea4648914c1e195 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sat, 20 Jun 2026 21:05:01 +0200 Subject: [PATCH 03/45] feat: email as a capture source (pipeline core) Email is one more ingestion source. Add a pure email_to_source mapper (body -> text, subject -> title, Message-ID -> an RFC 2392 mid: pointer for dedupe/ reprocess) and a capture_email entry that files it as an email-kind capture through the existing classify->mirror pipeline - so an email gets the same briefing + extracted action items as any document, no new intelligence. This is the pure, fully-tested core (no himalaya, no mailbox, no bot); it proves the ADR-010 seam that a new source slots into SourceContent. The mail container, stack mail CLI, mail bot + room routing follow. --- stacklets/docs/bot/capture_pipeline.py | 36 +++++++++++++++++++- stacklets/docs/bot/extractors.py | 28 ++++++++++++++++ tests/stacklets/test_capture_pipeline.py | 33 +++++++++++++++++++ tests/stacklets/test_extractors.py | 42 +++++++++++++++++++++++- 4 files changed, 137 insertions(+), 2 deletions(-) diff --git a/stacklets/docs/bot/capture_pipeline.py b/stacklets/docs/bot/capture_pipeline.py index 7bd141e..c2949dc 100644 --- a/stacklets/docs/bot/capture_pipeline.py +++ b/stacklets/docs/bot/capture_pipeline.py @@ -23,7 +23,7 @@ from loguru import logger from document_pipeline import utc_now_isoformat -from extractors import SourceContent +from extractors import SourceContent, email_to_source from matching import build_capture_event from notifier import Notifier from pdf_analysis import ( @@ -219,6 +219,40 @@ async def capture_text( bucket=bucket, ) + async def capture_email( + self, *, + subject: str | None, + body: str, + message_id: str | None, + sender_mxid: str, + from_addr: str | None = None, + bucket: str | None = None, + capture_id: str | None = None, + seed_topics: list[str] | None = None, + ) -> CaptureOutcome: + """File a fetched email as an `email`-kind capture. + + Reuses the capture pipeline wholesale: the body is the source the + classifier reads, the subject the title, the Message-ID the + dedupe/reprocess pointer (an RFC 2392 ``mid:`` URI). `from_addr` + (the sender) is surfaced to the classifier as a hint so it can + resolve the correspondent. Routing — which bucket, which room — is + the caller's job; here we just file. The `mail` container's + himalaya did the fetch; this is the in-pipeline handoff. + """ + source = email_to_source( + subject=subject, body=body, message_id=message_id, + ) + if not source.text.strip(): + return CaptureOutcome(status="empty") + user_hint = f"This is an email from {from_addr}." if from_addr else None + return await self._publish( + source=source, kind="email", sender_mxid=sender_mxid, + display_link=source.source_uri or "(email)", + user_hint=user_hint, actor=sender_mxid, + capture_id=capture_id, bucket=bucket, seed_topics=seed_topics, + ) + async def capture_voice_batch( self, *, transcripts: list[str], diff --git a/stacklets/docs/bot/extractors.py b/stacklets/docs/bot/extractors.py index 0ca2997..6271b0e 100644 --- a/stacklets/docs/bot/extractors.py +++ b/stacklets/docs/bot/extractors.py @@ -214,3 +214,31 @@ def _pick_title_hint(self, text: str) -> str | None: continue return line[: self._TITLE_HINT_MAX] return None + + +# ── Email ──────────────────────────────────────────────────────────────── +# +# Email is just another source. The `mail` container's himalaya has already +# fetched the message; this is the pure mapping from its parts into the +# classifier's `SourceContent`. No fetching, no I/O — fully testable on its +# own, and the seam ADR-010 wants every new source to slot into. + +def email_to_source( + *, subject: str | None, body: str, message_id: str | None = None, +) -> SourceContent: + """Map a fetched email into a `SourceContent`. + + The body is the text the classifier reads; the subject is the title + hint; the RFC822 Message-ID becomes the canonical pointer as an + RFC 2392 ``mid:`` URI — the stable key for dedupe and reprocess. + Angle brackets around the Message-ID are stripped; a blank subject or + a missing Message-ID collapse to ``None`` so a Dataview `where resource` + filters cleanly, same convention as the other capture sources. + """ + mid = (message_id or "").strip().strip("<>").strip() + return SourceContent( + text=body, + mime="text/plain", + title_hint=(subject or "").strip() or None, + source_uri=f"mid:{mid}" if mid else None, + ) diff --git a/tests/stacklets/test_capture_pipeline.py b/tests/stacklets/test_capture_pipeline.py index de89a36..8e6519e 100644 --- a/tests/stacklets/test_capture_pipeline.py +++ b/tests/stacklets/test_capture_pipeline.py @@ -1128,3 +1128,36 @@ async def test_seed_propagates_to_envelope(self): assert "camping" in envelope_tags + + +class TestCaptureEmail: + """Email is one more capture source: body → text, subject → title, + Message-ID → `mid:` pointer, filed as an `email`-kind capture through + the same pipeline as URLs and notes. Slice A1 of the email feature.""" + + @pytest.mark.asyncio + async def test_files_email_as_email_kind_capture(self): + mirror = FakeMirror() + pipe = _pipeline(mirror=mirror) + out = await pipe.capture_email( + subject="Elternabend am Freitag", + body="Bitte das Formular bis Freitag zurücksenden.", + message_id="", + sender_mxid="@homer:s", + from_addr="schule@example.org", + ) + assert out.status == "captured" + cap = mirror.captures[0] + assert cap["kind"] == "email" + assert cap["source_uri"] == "mid:abc123@school.example" + assert cap["title_hint"] == "Elternabend am Freitag" + + @pytest.mark.asyncio + async def test_empty_body_is_empty_outcome(self): + mirror = FakeMirror() + pipe = _pipeline(mirror=mirror) + out = await pipe.capture_email( + subject="x", body=" ", message_id="", sender_mxid="@homer:s", + ) + assert out.status == "empty" + assert mirror.captures == [] diff --git a/tests/stacklets/test_extractors.py b/tests/stacklets/test_extractors.py index e46c806..31201f0 100644 --- a/tests/stacklets/test_extractors.py +++ b/tests/stacklets/test_extractors.py @@ -21,7 +21,12 @@ _BOT_DIR = Path(__file__).resolve().parent.parent.parent / "stacklets" / "docs" / "bot" sys.path.insert(0, str(_BOT_DIR)) -from extractors import SourceContent, TextExtractor, UrlExtractor # noqa: E402 +from extractors import ( # noqa: E402 + SourceContent, + TextExtractor, + UrlExtractor, + email_to_source, +) # ── Fixture HTML ───────────────────────────────────────────────────────── @@ -271,3 +276,38 @@ async def test_url_alongside_text_stays_in_body(self): content = await TextExtractor().extract(body) assert "https://example.com/x" in content.text assert content.source_uri == "https://example.com/x" + + +# ── Email mapping ────────────────────────────────────────────────────────── + +class TestEmailToSource: + """`email_to_source` maps fetched email parts into SourceContent. + + Pure mapping, no I/O — the himalaya container already fetched the + message. The Message-ID becomes an RFC 2392 `mid:` pointer for dedupe + and reprocess.""" + + def test_maps_subject_body_and_message_id(self): + s = email_to_source( + subject="Elternabend am Freitag", + body="Bitte Formular zurücksenden.", + message_id="", + ) + assert isinstance(s, SourceContent) + assert s.text == "Bitte Formular zurücksenden." + assert s.title_hint == "Elternabend am Freitag" + assert s.source_uri == "mid:abc123@school.example" + + def test_strips_angle_brackets_from_message_id(self): + s = email_to_source(subject="x", body="y", message_id=" ") + assert s.source_uri == "mid:id@h" + + def test_blank_subject_is_none(self): + s = email_to_source(subject=" ", body="y", message_id="") + assert s.title_hint is None + + def test_missing_message_id_has_no_source_uri(self): + s = email_to_source(subject="x", body="y", message_id=None) + assert s.source_uri is None + s2 = email_to_source(subject="x", body="y", message_id="") + assert s2.source_uri is None From 02f0cfbf8b48a3bb1d4d03528e71ccbc7a54ef1f Mon Sep 17 00:00:00 2001 From: Arthur Date: Sun, 21 Jun 2026 14:00:16 +0200 Subject: [PATCH 04/45] feat: parse Maildir RFC822 emails for ingestion The mail bot reads himalaya's Maildir files (plain RFC822) directly rather than himalaya's rendered JSON: add parse_email (stdlib email, bytes in so UTF-8 bodies decode) extracting subject/from/message-id/date/body, the stable Message-ID included. capture_email now carries the email's real Date as captured_at. Pinned himalaya 1.2.0's actual contract against a fabricated local maildir: the README was wrong (flag is -o json not --json; envelope list is a bare array with from/to objects using addr and no message-id; maildir needs maildirpp=true). So ingestion stays on RFC822 + stdlib, not himalaya's output. --- docs/design/agent/email-tools.md | 47 ++++++++++++-------- stacklets/docs/bot/capture_pipeline.py | 3 +- stacklets/docs/bot/extractors.py | 60 ++++++++++++++++++++++++++ tests/stacklets/test_extractors.py | 57 ++++++++++++++++++++++++ 4 files changed, 147 insertions(+), 20 deletions(-) diff --git a/docs/design/agent/email-tools.md b/docs/design/agent/email-tools.md index b809b68..c2f74d1 100644 --- a/docs/design/agent/email-tools.md +++ b/docs/design/agent/email-tools.md @@ -84,25 +84,34 @@ Feed that into the existing `CapturePipeline` path (it already has `capture_*` sibling) and an email becomes a vault entry with a `> [!summary]` briefing and `## Action items` — no new intelligence written. -### himalaya JSON contract - -himalaya emits JSON with `--json`. **`envelope list --json`** is verified -against the himalaya README: - -```json -{"envelopes": [ - {"id": "...", "message-id": "...", "flags": [{"raw": "...", "iana": "..."}], - "subject": "...", "from": [{"name": "...", "email": "..."}], - "to": [...], "date": "...", "size": 0, "has-attachment": false}]} -``` - -Note `from`/`to` are **arrays of `{name, email}`**, not strings; field is -`message-id` (hyphen). **`message read --json`** (the body) is **not -documented in the README** — its exact shape must be pinned by running himalaya -once installed (Phase A2), not assumed. The code keeps the himalaya-JSON parser -isolated from the pure `email → SourceContent` mapper for exactly this reason: -the mapper's contract is famstack's own dataclass and is fully testable now; -the himalaya body schema is the one thing we confirm against the live binary. +### himalaya contract (pinned against v1.2.0, not the README) + +Pinned by running himalaya 1.2.0 against a fabricated local maildir. The README +was wrong on several points, so these are the real shapes: + +- **JSON flag is `-o json`** (`--output {plain,json}`), *not* `--json`. +- **Maildir backend needs `backend.maildirpp = true`** or it can't resolve INBOX. +- **`envelope list -o json`** is a *bare array* (not `{"envelopes": […]}`), and + carries **no Message-ID**: + ```json + [{"id":"2","flags":[],"subject":"…", + "from":{"name":"…","addr":"…"},"to":{"name":null,"addr":"…"}, + "date":"2026-06-20 10:00+00:00","has_attachment":false}] + ``` + `from`/`to` are single objects with **`addr`** (not arrays, not `email`); `id` + is a maildir sequence number, **not stable** across syncs. +- **`message read -o json`** returns a single JSON *string* of the rendered + message, body wrapped in himalaya MML (`<#part …>BODY<#/part>`). Awkward, and + there is no `--raw`. + +**Decision: ingestion does not parse himalaya's JSON at all.** himalaya's job is +IMAP↔Maildir transport (and SMTP send). The Maildir it produces is plain +**RFC822**, so the mail bot reads the message files directly with Python's +stdlib `email` module — clean Subject / From / Message-ID / Date / body, the +stable cross-sync key (Message-ID) included. himalaya's `envelope list -o json` +is used only by the human `stack mail list` CLI. This keeps the ingestion parser +on a standard (RFC822 + stdlib) instead of himalaya's rendering quirks, and it +is fully unit-testable with fabricated `.eml` strings. ## Architecture diff --git a/stacklets/docs/bot/capture_pipeline.py b/stacklets/docs/bot/capture_pipeline.py index c2949dc..0c84dd8 100644 --- a/stacklets/docs/bot/capture_pipeline.py +++ b/stacklets/docs/bot/capture_pipeline.py @@ -226,6 +226,7 @@ async def capture_email( message_id: str | None, sender_mxid: str, from_addr: str | None = None, + captured_at: str | None = None, bucket: str | None = None, capture_id: str | None = None, seed_topics: list[str] | None = None, @@ -249,7 +250,7 @@ async def capture_email( return await self._publish( source=source, kind="email", sender_mxid=sender_mxid, display_link=source.source_uri or "(email)", - user_hint=user_hint, actor=sender_mxid, + user_hint=user_hint, actor=sender_mxid, captured_at=captured_at, capture_id=capture_id, bucket=bucket, seed_topics=seed_topics, ) diff --git a/stacklets/docs/bot/extractors.py b/stacklets/docs/bot/extractors.py index 6271b0e..91070b6 100644 --- a/stacklets/docs/bot/extractors.py +++ b/stacklets/docs/bot/extractors.py @@ -27,6 +27,9 @@ import re from dataclasses import dataclass +from email import message_from_bytes +from email.policy import default as _email_policy +from email.utils import parseaddr, parsedate_to_datetime import aiohttp from loguru import logger @@ -242,3 +245,60 @@ def email_to_source( title_hint=(subject or "").strip() or None, source_uri=f"mid:{mid}" if mid else None, ) + + +@dataclass +class ParsedEmail: + """The fields the capture pipeline needs from one email message.""" + + subject: str | None + from_name: str | None + from_addr: str | None + message_id: str | None + date: str | None # captured_at, YYYY-MM-DD + body: str + + +def parse_email(raw: bytes) -> ParsedEmail: + """Parse an RFC822 message (a Maildir file's bytes) into a `ParsedEmail`. + + himalaya syncs IMAP into a Maildir of plain RFC822 files; the mail bot + reads those *as bytes* and parses here rather than via himalaya's + rendered JSON, so the contract is the standard email format + stdlib + `email` (modern `default` policy), not himalaya's output quirks. Bytes + in (not str) so declared charsets decode correctly — a UTF-8 body + round-trips. The plain-text body is preferred, falling back to HTML + content. Missing headers collapse to None / "" — the classifier copes. + """ + msg = message_from_bytes(raw, policy=_email_policy) + + from_name, from_addr = parseaddr(msg["from"] or "") + mid = (msg["message-id"] or "").strip().strip("<>").strip() or None + + captured_at = None + if msg["date"]: + try: + captured_at = parsedate_to_datetime(str(msg["date"])).date().isoformat() + except (TypeError, ValueError): + captured_at = None + + return ParsedEmail( + subject=(msg["subject"] or "").strip() or None, + from_name=(from_name or "").strip() or None, + from_addr=(from_addr or "").strip() or None, + message_id=mid, + date=captured_at, + body=_email_body(msg), + ) + + +def _email_body(msg) -> str: + """Best-effort plain-text body from an EmailMessage (default policy).""" + part = msg.get_body(preferencelist=("plain", "html")) + target = part if part is not None else msg + try: + content = target.get_content() + except (KeyError, LookupError): + payload = target.get_payload(decode=True) + content = payload.decode("utf-8", "replace") if isinstance(payload, bytes) else (payload or "") + return (content or "").strip() diff --git a/tests/stacklets/test_extractors.py b/tests/stacklets/test_extractors.py index 31201f0..12a09f3 100644 --- a/tests/stacklets/test_extractors.py +++ b/tests/stacklets/test_extractors.py @@ -26,6 +26,7 @@ TextExtractor, UrlExtractor, email_to_source, + parse_email, ) @@ -311,3 +312,59 @@ def test_missing_message_id_has_no_source_uri(self): assert s.source_uri is None s2 = email_to_source(subject="x", body="y", message_id="") assert s2.source_uri is None + + +# ── RFC822 parsing ───────────────────────────────────────────────────────── + +_EML = """From: Springfield School +To: family@example.org +Subject: Elternabend am Freitag +Message-ID: +Date: Fri, 20 Jun 2026 10:00:00 +0000 +Content-Type: text/plain; charset=utf-8 + +Bitte das Formular bis Freitag zurücksenden. +""" + + +class TestParseEmail: + """`parse_email` reads a Maildir RFC822 file with the stdlib email + module — the robust ingestion contract, independent of himalaya's + rendered output. Pinned against himalaya 1.2.0's Maildir files.""" + + def test_extracts_all_fields(self): + p = parse_email(_EML.encode("utf-8")) + assert p.subject == "Elternabend am Freitag" + assert p.from_name == "Springfield School" + assert p.from_addr == "office@springfield-school.example" + assert p.message_id == "abc123@springfield-school.example" + assert p.date == "2026-06-20" + assert p.body.strip() == "Bitte das Formular bis Freitag zurücksenden." + + def test_no_content_type_defaults_to_plain(self): + eml = ( + "From: a@b.example\nSubject: Hi\n" + "Message-ID: \n\nplain body here\n" + ) + p = parse_email(eml.encode("utf-8")) + assert p.body.strip() == "plain body here" + assert p.from_name is None # no display name + assert p.from_addr == "a@b.example" + + def test_missing_headers_collapse_to_none(self): + p = parse_email(b"\njust a body, no headers\n") + assert p.subject is None + assert p.message_id is None + assert p.date is None + assert "just a body" in p.body + + def test_multipart_prefers_plain(self): + eml = ( + "From: a@b.example\nSubject: multi\nMessage-ID: \n" + 'Content-Type: multipart/alternative; boundary="B"\n\n' + "--B\nContent-Type: text/plain\n\nthe plain part\n" + "--B\nContent-Type: text/html\n\n

the html part

\n--B--\n" + ) + p = parse_email(eml.encode("utf-8")) + assert "the plain part" in p.body + assert "

" not in p.body From 3842a8b25ff0b7c1627e7f6a25a69acd6fe1ef4e Mon Sep 17 00:00:00 2001 From: Arthur Date: Sun, 21 Jun 2026 15:29:37 +0200 Subject: [PATCH 05/45] docs: email vault paths, thread-folding, summary gate + transport finding - Paths: /emails/YYYY/MM/...; bucket from the room binding (family for a shared mailbox, person for a personal one); merge by date, account in frontmatter, no per-inbox subfolders. - Threading: one file per thread, keyed by the thread root and folded per ADR-010 (re-fetch + re-fold); each new message appends; distinguish an inbound reply (append) from a Matrix correction (rewrite). Thread-keying is day-one. - Summary gate: keep the body always; skip the LLM summary for short mail but still extract action items + tags. - Transport: the homebrew himalaya has no IMAP->Maildir sync, so sync is mbsync or stdlib imaplib behind the Maildir seam; himalaya stays for send + CLI. The seam makes the transport swappable, so himalaya's single-maintainer bus factor is not our exposure. --- docs/design/agent/email-tools.md | 68 ++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/docs/design/agent/email-tools.md b/docs/design/agent/email-tools.md index c2f74d1..327b116 100644 --- a/docs/design/agent/email-tools.md +++ b/docs/design/agent/email-tools.md @@ -113,6 +113,24 @@ is used only by the human `stack mail list` CLI. This keeps the ingestion parser on a standard (RFC822 + stdlib) instead of himalaya's rendering quirks, and it is fully unit-testable with fabricated `.eml` strings. +**Transport caveat — himalaya has no IMAP→Maildir sync in the homebrew build** +(`+imap +smtp +maildir`, no sync feature; `account sync` is absent). So himalaya +alone cannot populate the Maildir. Options, all behind the same Maildir seam: + +- **mbsync/isync** — the gold-standard IMAP→Maildir sync, in the `mail` container. + himalaya then handles only SMTP send + the `stack mail` CLI. +- **stdlib `imaplib`** — the bot fetches new messages by UID/Message-ID directly, + no external sync tool, no Maildir even. Simplest and most testable, but puts + IMAP creds + external network in bot-runner (less isolated than a `mail` + container). + +This is the swappable-transport property in action: the **Maildir (or just +RFC822 bytes) is the integration seam**; himalaya/mbsync/imaplib/msmtp are +interchangeable behind it, and `parse_email`/`capture_email` never change. +himalaya is mature (6.4k★, homebrew-core, active, v1.2) but single-maintainer; +the seam means that bus-factor is not our exposure. Decision between mbsync and +imaplib is deferred to the A2 build (the GreenMail round-trip informs it). + ## Architecture **Runtime: a container, not a host service.** himalaya needs no host resource @@ -203,6 +221,56 @@ says *which* room. Because the room is bound, a reply in it ("draft an answer", "remind me Friday") is scoped to that mailbox — the topic-rooms reply-chain pattern, reused. +## Vault layout, threading, and processing + +### Paths and bucket + +`/emails/YYYY/MM/-.md` — the capture shape (`/s/…`, +kind `email`), already produced by `capture_email`. The **bucket comes from the +room binding, not hardcoded**: a shared family mailbox files to `family/emails/…` +(institutional, like documents); a person's account files to `/emails/…` +(personal, like their captures). Default = the bucket that owns the account. + +**Merge by date, do not subfolder per inbox.** Account + folder live in +frontmatter (and the room binding), not as a path level — consistent with the +rest of the vault, where identity is frontmatter + date path and the channel is +never a directory. Two inboxes to the same bucket merge on disk but stay distinct +rooms + distinct frontmatter; dedup by Message-ID keeps it safe. + +### Threading: one file per thread, folded (per ADR-010) + +An email thread *is* a reply chain, and [adr-010](../adr/adr-010-event-pipeline.md) +already says a filing's source is the whole thread and the vault entry is the +*fold* of it. So: + +- **Key the entry by the thread root, not the message** — resolved from + `References` / `In-Reply-To` (root Message-ID). Get this right up front; it + changes `capture_email`'s identity model and is painful to retrofit. +- **Each new message updates the same file** — appends the message, re-renders + the conversation chronologically, refreshes a thread-level summary + the + accumulated action items. +- **Reproducible**: re-fetch every message in the thread by Message-ID, re-fold. +- **Append vs. correct** — both are "reply chains" but mean different things: a + real inbound email *appends* to the conversation; a Matrix reply from a family + member *corrects* the filing (rewrites). The sender / Message-ID distinguishes + them. The thread folder must not treat a human correction as a new email. + +Staging: A2 ships per-message append into the thread file; the polished thread +summary + folded action items can iterate. Thread-keying is day-one. + +### Summary gate for short mail + +Email runs the same classifier as everything else (title, summary, facts, +action_items, tags). But a two-line mail does not need an LLM summary — it would +be longer than the mail and waste a model call. So: + +- **Always keep the email body** (unlike a bookmark; the body *is* the content). +- **Gate the summary on length** — below ~a few hundred chars, skip the + `> [!summary]` briefing; the body stands alone. +- **Still extract action_items + tags when short** — cheap and the highest-value + bit ("send the form back Friday" is short but actionable). Short ≠ no + processing; short = *no summary*. + ## What we reuse vs. build **Reuse (no new code):** From 894cec6dca7d3a17de151fbb32f14e0a637b6b67 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sun, 21 Jun 2026 15:55:08 +0200 Subject: [PATCH 06/45] test: pin email IMAP ingestion end-to-end against GreenMail Spins up a throwaway GreenMail (auth-disabled, fabricated data), sends via SMTP, fetches with stdlib imaplib, and asserts parse_email round-trips the message faithfully - including a UTF-8 body. Proves the chosen ingestion transport (imaplib + parse_email, no himalaya/mbsync) against a real IMAP server. Self -contained: manages and tears down its own container; skipped without Docker. --- tests/integration/test_email_imap_e2e.py | 110 +++++++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100644 tests/integration/test_email_imap_e2e.py diff --git a/tests/integration/test_email_imap_e2e.py b/tests/integration/test_email_imap_e2e.py new file mode 100644 index 0000000..237e917 --- /dev/null +++ b/tests/integration/test_email_imap_e2e.py @@ -0,0 +1,110 @@ +"""End-to-end: a real IMAP server (GreenMail) -> imaplib -> parse_email. + +Pins the email ingestion transport against a real IMAP server with fabricated +data only (no real account, no network). The chosen ingestion path is stdlib +`imaplib` fetch + `parse_email`; this proves it round-trips a UTF-8 message +faithfully end to end, which a unit test on fixture strings cannot. + +Self-contained: spins up and tears down its own GreenMail container, so it +leaves nothing behind. Skipped when Docker is unavailable. +""" + +from __future__ import annotations + +import imaplib +import shutil +import smtplib +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +_BOT_DIR = Path(__file__).resolve().parent.parent.parent / "stacklets" / "docs" / "bot" +sys.path.insert(0, str(_BOT_DIR)) + +from extractors import parse_email # noqa: E402 + +pytestmark = pytest.mark.skipif( + shutil.which("docker") is None, reason="docker not available", +) + +_IMAGE = "greenmail/standalone:2.1.3" +_NAME = "famstack-greenmail-pytest" +_SMTP_PORT = 3025 +_IMAP_PORT = 3143 + + +@pytest.fixture(scope="module") +def greenmail(): + """Run a throwaway GreenMail (auth disabled, test ports), tear it down.""" + subprocess.run(["docker", "rm", "-f", _NAME], capture_output=True) + started = subprocess.run( + [ + "docker", "run", "-d", "--name", _NAME, + "-p", f"{_SMTP_PORT}:3025", "-p", f"{_IMAP_PORT}:3143", + "-e", ( + "GREENMAIL_OPTS=-Dgreenmail.setup.test.all " + "-Dgreenmail.hostname=0.0.0.0 -Dgreenmail.auth.disabled" + ), + _IMAGE, + ], + capture_output=True, text=True, + ) + if started.returncode != 0: + pytest.skip(f"could not start GreenMail: {started.stderr.strip()}") + try: + deadline = time.time() + 30 + while time.time() < deadline: + try: + imaplib.IMAP4("localhost", _IMAP_PORT).logout() + break + except (OSError, EOFError, imaplib.IMAP4.error): + # connection refused, or socket up but banner not ready yet + time.sleep(0.5) + else: + pytest.skip("GreenMail IMAP did not come up in time") + yield + finally: + subprocess.run(["docker", "rm", "-f", _NAME], capture_output=True) + + +def test_imap_roundtrip_parses_faithfully(greenmail): + user = "homer@example.org" + raw = ( + "From: Springfield School \r\n" + f"To: {user}\r\n" + "Subject: Elternabend am Freitag\r\n" + "Message-ID: \r\n" + "Date: Sat, 21 Jun 2026 09:00:00 +0000\r\n" + "Content-Type: text/plain; charset=utf-8\r\n" + "\r\n" + "Bitte das Formular bis Freitag zurücksenden.\r\n" + ).encode("utf-8") + + # Send through GreenMail's SMTP. + smtp = smtplib.SMTP("localhost", _SMTP_PORT, timeout=10) + smtp.sendmail("office@springfield-school.example", [user], raw) + smtp.quit() + time.sleep(1) + + # Fetch through GreenMail's IMAP with the stdlib client (the ingestion path). + imap = imaplib.IMAP4("localhost", _IMAP_PORT) + imap.login(user, "irrelevant") # auth disabled + imap.select("INBOX") + _, data = imap.search(None, "ALL") + ids = data[0].split() + assert ids, "message was not delivered to INBOX" + _, msgdata = imap.fetch(ids[-1], "(RFC822)") + fetched = msgdata[0][1] + imap.logout() + + # The full ingestion contract, end to end against a real IMAP server. + p = parse_email(fetched) + assert p.subject == "Elternabend am Freitag" + assert p.from_name == "Springfield School" + assert p.from_addr == "office@springfield-school.example" + assert p.message_id == "e2e-1@springfield-school.example" + assert p.date == "2026-06-21" + assert p.body.strip() == "Bitte das Formular bis Freitag zurücksenden." From b6b9bf3b025e38445547b981fd5520c31774d920 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sun, 21 Jun 2026 16:39:37 +0200 Subject: [PATCH 07/45] feat: IMAP mail fetcher with thread-keying and dedup parse_email now resolves the thread root (References[0], else In-Reply-To, else own Message-ID) - the day-one identity every message in a conversation folds into per ADR-010. Add MailFetcher: read-only stdlib imaplib fetch of new messages, deduped by Message-ID against a caller-supplied seen set so re-runs are idempotent (blocking I/O, the bot calls it via asyncio.to_thread). Verified against GreenMail: fetch + dedup + idempotent re-run. --- stacklets/docs/bot/extractors.py | 33 +++++++++- stacklets/docs/bot/mail_fetcher.py | 81 ++++++++++++++++++++++++ tests/integration/test_email_imap_e2e.py | 33 ++++++++++ tests/stacklets/test_extractors.py | 32 ++++++++++ 4 files changed, 178 insertions(+), 1 deletion(-) create mode 100644 stacklets/docs/bot/mail_fetcher.py diff --git a/stacklets/docs/bot/extractors.py b/stacklets/docs/bot/extractors.py index 91070b6..95faeaa 100644 --- a/stacklets/docs/bot/extractors.py +++ b/stacklets/docs/bot/extractors.py @@ -26,7 +26,7 @@ from __future__ import annotations import re -from dataclasses import dataclass +from dataclasses import dataclass, field from email import message_from_bytes from email.policy import default as _email_policy from email.utils import parseaddr, parsedate_to_datetime @@ -257,6 +257,24 @@ class ParsedEmail: message_id: str | None date: str | None # captured_at, YYYY-MM-DD body: str + references: list[str] = field(default_factory=list) + in_reply_to: str | None = None + + @property + def thread_root(self) -> str | None: + """The Message-ID the conversation is keyed by. + + A reply carries `References` (ancestors, oldest first); its first + entry is the thread root. Failing that, the immediate parent + (`In-Reply-To`). A message that starts a thread has neither, so it + is its own root. This is the vault entry's identity — every message + in a conversation folds into the file keyed by this id (ADR-010). + """ + if self.references: + return self.references[0] + if self.in_reply_to: + return self.in_reply_to + return self.message_id def parse_email(raw: bytes) -> ParsedEmail: @@ -282,6 +300,7 @@ def parse_email(raw: bytes) -> ParsedEmail: except (TypeError, ValueError): captured_at = None + in_reply = _parse_msgids(msg["in-reply-to"]) return ParsedEmail( subject=(msg["subject"] or "").strip() or None, from_name=(from_name or "").strip() or None, @@ -289,9 +308,21 @@ def parse_email(raw: bytes) -> ParsedEmail: message_id=mid, date=captured_at, body=_email_body(msg), + references=_parse_msgids(msg["references"]), + in_reply_to=in_reply[0] if in_reply else None, ) +_MSGID_RE = re.compile(r"<([^<>]+)>") + + +def _parse_msgids(header) -> list[str]: + """Bare Message-IDs from a References / In-Reply-To header (angle-bracketed).""" + if not header: + return [] + return [m.strip() for m in _MSGID_RE.findall(str(header)) if m.strip()] + + def _email_body(msg) -> str: """Best-effort plain-text body from an EmailMessage (default policy).""" part = msg.get_body(preferencelist=("plain", "html")) diff --git a/stacklets/docs/bot/mail_fetcher.py b/stacklets/docs/bot/mail_fetcher.py new file mode 100644 index 0000000..000440e --- /dev/null +++ b/stacklets/docs/bot/mail_fetcher.py @@ -0,0 +1,81 @@ +"""IMAP fetcher — pulls new email off the server for ingestion. + +Stdlib `imaplib` is the chosen ingestion transport (himalaya/mbsync are +swappable behind the same RFC822 seam; see docs/design/agent/email-tools.md). +It is blocking I/O, so the async mail bot calls `fetch_new` via +`asyncio.to_thread`, exactly as the git mirror wraps the sync Forgejo client. + +New-message detection is by Message-ID against a caller-supplied seen set, so a +re-run never double-files (ADR-010 dedup). Ingestion is read-only: the folder is +selected `readonly=True`, so server flags are never mutated. +""" + +from __future__ import annotations + +import imaplib +from dataclasses import dataclass + +from extractors import ParsedEmail, parse_email + + +@dataclass +class MailAccount: + """Connection details for one IMAP account + folder.""" + + host: str + port: int + user: str + password: str + folder: str = "INBOX" + ssl: bool = True + timeout: int = 30 + + +class MailFetcher: + """Read-only IMAP fetch of new messages, parsed for the pipeline.""" + + def __init__(self, account: MailAccount): + self._a = account + + def fetch_new(self, seen_message_ids: set[str]) -> list[ParsedEmail]: + """Parsed messages in the folder whose Message-ID is not yet seen. + + Connects, selects the folder read-only, fetches each message's RFC822 + bytes, parses, and drops any whose Message-ID is already in + ``seen_message_ids``. The caller persists the seen set (the bot's + cache), so re-running is idempotent. Messages with no Message-ID can't + be deduped and are always returned — vanishingly rare in real mail. + """ + client = self._connect() + try: + typ, _ = client.select(self._a.folder, readonly=True) + if typ != "OK": + return [] + typ, data = client.search(None, "ALL") + if typ != "OK" or not data or not data[0]: + return [] + + out: list[ParsedEmail] = [] + for num in data[0].split(): + typ, msgdata = client.fetch(num, "(RFC822)") + if typ != "OK" or not msgdata or not msgdata[0]: + continue + raw = msgdata[0][1] + if not isinstance(raw, (bytes, bytearray)): + continue + parsed = parse_email(bytes(raw)) + if parsed.message_id and parsed.message_id in seen_message_ids: + continue + out.append(parsed) + return out + finally: + try: + client.logout() + except (imaplib.IMAP4.error, OSError): + pass + + def _connect(self) -> imaplib.IMAP4: + cls = imaplib.IMAP4_SSL if self._a.ssl else imaplib.IMAP4 + client = cls(self._a.host, self._a.port, timeout=self._a.timeout) + client.login(self._a.user, self._a.password) + return client diff --git a/tests/integration/test_email_imap_e2e.py b/tests/integration/test_email_imap_e2e.py index 237e917..00f5b7e 100644 --- a/tests/integration/test_email_imap_e2e.py +++ b/tests/integration/test_email_imap_e2e.py @@ -25,6 +25,7 @@ sys.path.insert(0, str(_BOT_DIR)) from extractors import parse_email # noqa: E402 +from mail_fetcher import MailAccount, MailFetcher # noqa: E402 pytestmark = pytest.mark.skipif( shutil.which("docker") is None, reason="docker not available", @@ -108,3 +109,35 @@ def test_imap_roundtrip_parses_faithfully(greenmail): assert p.message_id == "e2e-1@springfield-school.example" assert p.date == "2026-06-21" assert p.body.strip() == "Bitte das Formular bis Freitag zurücksenden." + + +def _send(user: str, message_id: str, subject: str) -> None: + raw = ( + f"From: Sender \r\nTo: {user}\r\n" + f"Subject: {subject}\r\nMessage-ID: {message_id}\r\n" + "Date: Sat, 21 Jun 2026 09:00:00 +0000\r\n\r\nbody\r\n" + ).encode("utf-8") + smtp = smtplib.SMTP("localhost", _SMTP_PORT, timeout=10) + smtp.sendmail("sender@x.example", [user], raw) + smtp.quit() + + +def test_mailfetcher_fetches_new_and_dedups(greenmail): + user = "marge@example.org" # own INBOX, isolated from the other test + _send(user, "", "First") + _send(user, "", "Second") + time.sleep(1) + + fetcher = MailFetcher(MailAccount( + host="localhost", port=_IMAP_PORT, user=user, password="x", ssl=False, + )) + + first = fetcher.fetch_new(set()) + assert {p.message_id for p in first} == {"m1@h", "m2@h"} + + # Already-seen Message-IDs are dropped -> idempotent re-run. + again = fetcher.fetch_new({"m1@h"}) + assert {p.message_id for p in again} == {"m2@h"} + + nothing = fetcher.fetch_new({"m1@h", "m2@h"}) + assert nothing == [] diff --git a/tests/stacklets/test_extractors.py b/tests/stacklets/test_extractors.py index 12a09f3..f077755 100644 --- a/tests/stacklets/test_extractors.py +++ b/tests/stacklets/test_extractors.py @@ -368,3 +368,35 @@ def test_multipart_prefers_plain(self): p = parse_email(eml.encode("utf-8")) assert "the plain part" in p.body assert "

" not in p.body + + +class TestThreadRoot: + """A thread folds into one vault entry keyed by its root Message-ID + (ADR-010). References[0] is the root; else In-Reply-To; else the + message starts its own thread.""" + + def test_standalone_message_is_its_own_root(self): + p = parse_email(b"Message-ID: \nSubject: x\n\nbody\n") + assert p.references == [] + assert p.in_reply_to is None + assert p.thread_root == "solo@h" + + def test_references_first_entry_is_root(self): + eml = ( + b"Message-ID: \n" + b"In-Reply-To: \n" + b"References: \n" + b"Subject: Re: x\n\nlater\n" + ) + p = parse_email(eml) + assert p.references == ["root@h", "reply1@h"] + assert p.in_reply_to == "reply1@h" + assert p.thread_root == "root@h" + + def test_in_reply_to_when_no_references(self): + eml = ( + b"Message-ID: \nIn-Reply-To: \n" + b"Subject: Re: x\n\nfirst reply\n" + ) + p = parse_email(eml) + assert p.thread_root == "root@h" From fe30bddb6ae6729008d450161d7615ecee23328e Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 22 Jun 2026 18:03:33 +0200 Subject: [PATCH 08/45] feat: fold an email conversation into one thread file Every message in a conversation now lands in a single vault entry keyed by the thread root, instead of one file per message. A reply appends a dated section; persons and tags union across the thread so indexing spans the whole exchange. Re-filing a message already in the file is a no-op (each section carries a mid: marker, so idempotency lives in the file). Email keeps action items (a school's 'return the form by Friday' is a task worth surfacing) unlike bookmarks, which drop them. --- stacklets/docs/bot/capture_pipeline.py | 82 ++++++---- stacklets/docs/bot/extractors.py | 15 +- stacklets/docs/bot/git_mirror.py | 134 ++++++++++++++++ stacklets/docs/bot/vault_entry.py | 193 +++++++++++++++++++++++ tests/stacklets/test_capture_pipeline.py | 54 ++++++- tests/stacklets/test_extractors.py | 20 +-- tests/stacklets/test_vault_entry.py | 187 ++++++++++++++++++++++ 7 files changed, 632 insertions(+), 53 deletions(-) diff --git a/stacklets/docs/bot/capture_pipeline.py b/stacklets/docs/bot/capture_pipeline.py index 0c84dd8..a15e850 100644 --- a/stacklets/docs/bot/capture_pipeline.py +++ b/stacklets/docs/bot/capture_pipeline.py @@ -225,24 +225,28 @@ async def capture_email( body: str, message_id: str | None, sender_mxid: str, + thread_root: str | None = None, from_addr: str | None = None, captured_at: str | None = None, bucket: str | None = None, capture_id: str | None = None, seed_topics: list[str] | None = None, ) -> CaptureOutcome: - """File a fetched email as an `email`-kind capture. - - Reuses the capture pipeline wholesale: the body is the source the - classifier reads, the subject the title, the Message-ID the - dedupe/reprocess pointer (an RFC 2392 ``mid:`` URI). `from_addr` - (the sender) is surfaced to the classifier as a hint so it can - resolve the correspondent. Routing — which bucket, which room — is - the caller's job; here we just file. The `mail` container's - himalaya did the fetch; this is the in-pipeline handoff. + """Fold a fetched email into its thread file. + + Reuses the capture pipeline's classify/tag/envelope tail wholesale: + the body is the source the classifier reads, the subject the title, + the *thread root* the file identity (an RFC 2392 ``mid:`` URI). The + write differs from a URL/note capture — email accumulates, so each + message folds into one thread file (see `publish_email_message`) + rather than replacing a single-shot entry. `from_addr` is surfaced + to the classifier as a hint so it can resolve the correspondent, + and stamps the message's section heading. Routing — which bucket, + which room — is the caller's job; here we just fold. The `mail` + container did the fetch; this is the in-pipeline handoff. """ source = email_to_source( - subject=subject, body=body, message_id=message_id, + subject=subject, body=body, thread_root=thread_root or message_id, ) if not source.text.strip(): return CaptureOutcome(status="empty") @@ -252,6 +256,7 @@ async def capture_email( display_link=source.source_uri or "(email)", user_hint=user_hint, actor=sender_mxid, captured_at=captured_at, capture_id=capture_id, bucket=bucket, seed_topics=seed_topics, + email_meta={"message_id": message_id, "from_addr": from_addr}, ) async def capture_voice_batch( @@ -635,6 +640,7 @@ async def _publish( initial_classification: dict | None = None, seed_topics: list[str] | None = None, bucket: str | None = None, + email_meta: dict | None = None, ) -> CaptureOutcome: """Shared tail: classify, mirror, record tags, return the outcome. @@ -676,24 +682,44 @@ async def _publish( tags = self._tag_list(classification) captured_at = captured_at or _dt.date.today().isoformat() - # Bookmarks default to marker mode (body dropped, summary IS the - # content); notes always keep the body. - keep_body = kind == "note" or self.capture_keep_body - body_for_mirror = source.text if keep_body else "" - - vault_path = await self._mirror.publish_capture( - entity=entity_slug, - kind=kind, - source_uri=source.source_uri, - title_hint=source.title_hint, - body_text=body_for_mirror, - classification=classification, - captured_at=captured_at, - model=model, - tags=tags, - existing_path=existing_path, - capture_id=capture_id, - ) + # Email folds into a thread file (append a dated section) instead + # of replacing a single-shot entry; `email_meta` carries the + # per-message identity (Message-ID, sender) the fold needs. + # Everything else above — classify, tags, the envelope below — is + # shared with URL/note captures. Body is always kept: the + # conversation IS the content. + if kind == "email" and email_meta is not None: + vault_path = await self._mirror.publish_email_message( + entity=entity_slug, + thread_uri=source.source_uri, + message_id=email_meta.get("message_id"), + from_addr=email_meta.get("from_addr"), + title_hint=source.title_hint, + body_text=source.text, + classification=classification, + captured_at=captured_at, + model=model, + tags=tags, + capture_id=capture_id, + ) + else: + # Bookmarks default to marker mode (body dropped, summary IS + # the content); notes always keep the body. + keep_body = kind == "note" or self.capture_keep_body + body_for_mirror = source.text if keep_body else "" + vault_path = await self._mirror.publish_capture( + entity=entity_slug, + kind=kind, + source_uri=source.source_uri, + title_hint=source.title_hint, + body_text=body_for_mirror, + classification=classification, + captured_at=captured_at, + model=model, + tags=tags, + existing_path=existing_path, + capture_id=capture_id, + ) # Feed topic tags (not the derived Person: X) back into the # vocabulary cache so the next capture's prompt sees them. diff --git a/stacklets/docs/bot/extractors.py b/stacklets/docs/bot/extractors.py index 95faeaa..bb6c193 100644 --- a/stacklets/docs/bot/extractors.py +++ b/stacklets/docs/bot/extractors.py @@ -227,18 +227,19 @@ def _pick_title_hint(self, text: str) -> str | None: # own, and the seam ADR-010 wants every new source to slot into. def email_to_source( - *, subject: str | None, body: str, message_id: str | None = None, + *, subject: str | None, body: str, thread_root: str | None = None, ) -> SourceContent: """Map a fetched email into a `SourceContent`. The body is the text the classifier reads; the subject is the title - hint; the RFC822 Message-ID becomes the canonical pointer as an - RFC 2392 ``mid:`` URI — the stable key for dedupe and reprocess. - Angle brackets around the Message-ID are stripped; a blank subject or - a missing Message-ID collapse to ``None`` so a Dataview `where resource` - filters cleanly, same convention as the other capture sources. + hint; the *thread root* Message-ID becomes the canonical pointer as an + RFC 2392 ``mid:`` URI. It keys the thread file, not the individual + message — every reply folds into the same entry (ADR-010). Angle + brackets are stripped; a blank subject or a missing thread root + collapse to ``None`` so a Dataview `where resource` filters cleanly, + same convention as the other capture sources. """ - mid = (message_id or "").strip().strip("<>").strip() + mid = (thread_root or "").strip().strip("<>").strip() return SourceContent( text=body, mime="text/plain", diff --git a/stacklets/docs/bot/git_mirror.py b/stacklets/docs/bot/git_mirror.py index 3609ec5..a4ad1db 100644 --- a/stacklets/docs/bot/git_mirror.py +++ b/stacklets/docs/bot/git_mirror.py @@ -62,6 +62,8 @@ capture_frontmatter, render_document, render_capture, + render_email_message_section, + fold_email_message, ) @@ -773,3 +775,135 @@ async def publish_capture( logger.info("[git-mirror] {} → {}", verb, target_path) return target_path + + # ── Email threads ───────────────────────────────────────────────────── + # + # Email is the one capture source that accumulates: every message in a + # conversation folds into a single file keyed by the thread root, so the + # whole exchange reads top to bottom in one place. New messages append a + # dated section; re-folding a message already present is a no-op (its + # `mid:` marker is detected). The path lives under the routed bucket: + # + # /emails/YYYY/MM/-.md + # + # The shape (frontmatter, sections, markers) is owned by vault_entry; + # this method is just the Forgejo read-fold-write around it. + + async def publish_email_message( + self, *, + entity: str, + thread_uri: str | None, + message_id: str | None, + from_addr: str | None, + title_hint: str | None, + body_text: str, + classification: dict, + captured_at: str, + model: str | None, + tags: list[str] | None = None, + capture_id: str | None = None, + ) -> str | None: + """Fold one email message into its thread file. + + `entity` is the routed bucket (mailbox binding) or the sender's + slug. `thread_uri` is the ``mid:`` URI of the thread root — it keys + the file, so every reply lands in the same entry. Returns the path + the thread lives at (even when the message was already folded, an + idempotent no-op), or None when Forgejo is unreachable or the write + failed. Best-effort like the rest of the mirror. + """ + if not await self.ensure_setup(): + return None + + client = ForgejoClient(url=self.code_url, token=self._creds.token) + + resolved_title = classification.get("title") or title_hint + title = resolved_title or "Email thread" + + persons_raw = classification.get("persons") or classification.get("person") or [] + if isinstance(persons_raw, str): + persons_raw = [persons_raw] + persons = [p for p in persons_raw if isinstance(p, str) and p] + + target_path = self._capture_filepath( + entity=entity, + kind="email", + captured_at=captured_at, + title=title if resolved_title else None, + hash_key=thread_uri or body_text, + ) + + existing = await asyncio.to_thread( + client.get_file, self.repo_owner, REPO_NAME, target_path, + ) + existing_content = (existing.get("content") if existing else None) or None + + section = render_email_message_section( + message_id=message_id, + from_addr=from_addr, + captured_at=captured_at, + body=body_text, + summary=classification.get("summary"), + facts=classification.get("facts") or [], + action_items=classification.get("action_items") or [], + ) + + new_fm = self._capture_frontmatter( + title=title, + captured_at=captured_at, + kind="email", + source_uri=thread_uri, + persons=persons, + tags=tags or [], + model=model, + capture_id=capture_id, + ) + + content = fold_email_message( + existing_content, + section=section, + message_id=message_id, + new_frontmatter=new_fm, + title=title, + captured_at=captured_at, + source_uri=thread_uri, + persons=persons, + tags=tags or [], + from_path=target_path, + shared_bucket=self.shared_bucket, + ) + if content is None: + # Already folded into this thread — nothing to write. + logger.info("[git-mirror] email already folded → {}", target_path) + return target_path + + verb = "update" if existing_content else "capture" + summary = classification.get("summary") + message_lines = [f"{verb}: {title}", ""] + if summary: + message_lines.append(summary.strip()) + message_lines.append("") + message_lines.append(f"Thread: {thread_uri or '(none)'}") + if message_id: + message_lines.append(f"Message-Id: {message_id}") + if model: + message_lines.append(f"Model: {model}") + commit = "\n".join(message_lines) + + try: + await asyncio.to_thread( + client.put_file, + self.repo_owner, REPO_NAME, target_path, + content=content, message=commit, + sha=existing["sha"] if existing else None, + author_name=BOT_USERNAME, author_email=BOT_EMAIL, + ) + except ForgejoError as e: + logger.warning( + "[git-mirror] Email fold failed for {}: {}", + thread_uri or "(no thread)", e, + ) + return None + + logger.info("[git-mirror] {} email → {}", verb, target_path) + return target_path diff --git a/stacklets/docs/bot/vault_entry.py b/stacklets/docs/bot/vault_entry.py index ae2c04b..5d89540 100644 --- a/stacklets/docs/bot/vault_entry.py +++ b/stacklets/docs/bot/vault_entry.py @@ -487,6 +487,199 @@ def render_capture( return "\n".join(parts) +# ── Email threads ────────────────────────────────────────────────────── +# +# Email is the one capture source that accumulates. A URL or a note is a +# single artifact; a conversation is many messages that belong together, +# so they fold into ONE file keyed by the thread root Message-ID (the +# identity ADR-010 reprocessing replays): +# +# /emails/YYYY/MM/-.md +# +# where is over the thread root, so every reply resolves to the +# same path. The file is a stack of dated message sections under a shared +# H1 + frontmatter. Each section carries an HTML-comment `mid:` marker so +# re-folding the same message is a no-op — idempotency lives in the file +# itself, not in external state (the vault is the source of truth). +# +# Unlike bookmarks, email keeps action items: "return the form by Friday" +# is exactly the kind of task the brain should surface. + + +def email_mid_marker(message_id: str) -> str: + """The per-section idempotency marker for one message in a thread.""" + return f"" + + +def render_email_message_section( + *, + message_id: str | None, + from_addr: str | None, + captured_at: str | None, + body: str, + summary: str | None = None, + facts: list | None = None, + action_items: list | None = None, +) -> str: + """Render one message as a dated section of its thread file. + + Layout: the idempotency marker comment, an H2 heading (date — sender), + the per-message briefing callout (summary, facts, action items), then + the verbatim message body in a collapsible quote callout. Trailing + newline so sections concatenate with a blank line between them. + """ + parts: list[str] = [] + if message_id: + parts.append(email_mid_marker(message_id)) + heading_bits = [b for b in (captured_at, from_addr) if b] + parts.append("## " + (" — ".join(heading_bits) or "Message")) + parts.append("") + + briefing = _briefing_block( + summary=summary, facts=facts, action_items=action_items, + ) + if briefing: + parts.append(briefing) + parts.append("") + + body_stripped = body.strip() if body else "" + if body_stripped: + parts.append("> [!quote]- Message") + for ln in body_stripped.split("\n"): + parts.append(f"> {ln}" if ln else ">") + return "\n".join(parts).rstrip() + "\n" + + +def render_email_thread( + *, + frontmatter: dict, + title: str, + captured_at: str | None, + source_uri: str | None, + persons: list[str], + from_path: str, + shared_bucket: str, + sections: list[str], +) -> str: + """Assemble a fresh thread file: frontmatter + H1 + meta + sections.""" + fm_yaml = yaml.safe_dump( + frontmatter, sort_keys=False, allow_unicode=True, default_flow_style=False, + ).strip() + parts = ["---", fm_yaml, "---", "", f"# {title}", ""] + + meta_lines: list[str] = [] + if persons: + meta_lines.append( + "**About** " + ", ".join( + f"[{p}]({entity_relpath(p, 'person', from_path, shared_bucket)})" + for p in persons + ) + ) + line2 = [] + if captured_at: + line2.append(f"**Started** {captured_at}") + line2.append("**Kind** email") + meta_lines.append(" · ".join(line2)) + if source_uri: + meta_lines.append(f"**Thread** <{source_uri}>") + parts.extend(f"> {ln}" for ln in meta_lines) + + for section in sections: + parts.append("") + parts.append(section.rstrip()) + return "\n".join(parts).rstrip() + "\n" + + +def split_frontmatter(content: str) -> tuple[dict, str]: + """Split a markdown file into (frontmatter dict, body after frontmatter). + + Returns ``({}, content)`` when there is no leading ``---`` block, so a + malformed or frontmatter-less file degrades to "all body" rather than + raising. + """ + if not content.startswith("---\n"): + return ({}, content) + end = content.find("\n---\n", 4) + if end < 0: + return ({}, content) + try: + fm = yaml.safe_load(content[4:end]) + except yaml.YAMLError: + fm = None + body = content[end + len("\n---\n"):] + return (fm if isinstance(fm, dict) else {}, body) + + +def merge_email_frontmatter( + old: dict, *, persons: list[str], tags: list[str], +) -> dict: + """Union a new message's persons and tags into the thread frontmatter. + + A reply can loop in a new person or surface a new topic; the thread + file's frontmatter is the union across every folded message so person- + and tag-indexing span the whole conversation. Existing values keep + their order, new ones append, duplicates collapse. + """ + fm = dict(old) + for key, incoming in (("persons", persons), ("tags", tags)): + if not incoming: + continue + merged = list(fm.get(key) or []) + for v in incoming: + if v and v not in merged: + merged.append(v) + if merged: + fm[key] = merged + return fm + + +def fold_email_message( + existing_content: str | None, + *, + section: str, + message_id: str | None, + new_frontmatter: dict, + title: str, + captured_at: str | None, + source_uri: str | None, + persons: list[str], + tags: list[str], + from_path: str, + shared_bucket: str, +) -> str | None: + """Fold one message into its thread file; return the new file content. + + Three cases: + - First message of a thread (no existing file) → render the shell + plus this section. + - Already-folded message (its ``mid:`` marker is in the file) → + return ``None`` so the caller skips the write (idempotent). + - A new message in an existing thread → append the section and union + its persons/tags into the existing frontmatter. + """ + if not existing_content: + return render_email_thread( + frontmatter=new_frontmatter, title=title, + captured_at=captured_at, source_uri=source_uri, + persons=persons, from_path=from_path, + shared_bucket=shared_bucket, sections=[section], + ) + if message_id and email_mid_marker(message_id) in existing_content: + return None + fm_old, body_old = split_frontmatter(existing_content) + fm_merged = merge_email_frontmatter(fm_old, persons=persons, tags=tags) + fm_yaml = yaml.safe_dump( + fm_merged, sort_keys=False, allow_unicode=True, default_flow_style=False, + ).strip() + return ( + f"---\n{fm_yaml}\n---\n" + + body_old.rstrip() + + "\n\n" + + section.rstrip() + + "\n" + ) + + # ── Briefing block ─────────────────────────────────────────────────── # # The briefing is the classifier's per-document take, rendered as an diff --git a/tests/stacklets/test_capture_pipeline.py b/tests/stacklets/test_capture_pipeline.py index 8e6519e..089409c 100644 --- a/tests/stacklets/test_capture_pipeline.py +++ b/tests/stacklets/test_capture_pipeline.py @@ -57,6 +57,7 @@ async def reformat(self, ocr_text: str, *, max_chars: int = 20000): class FakeMirror: def __init__(self): self.captures: list[dict] = [] + self.emails: list[dict] = [] async def publish_capture(self, **kwargs): self.captures.append(kwargs) @@ -67,6 +68,13 @@ async def publish_capture(self, **kwargs): entity = "homer" return f"{entity}/{kind}s/test-capture.md" + async def publish_email_message(self, **kwargs): + # Email folds into a thread file via its own mirror entrypoint; + # the path is keyed by the thread, not the individual message. + self.emails.append(kwargs) + entity = kwargs.get("entity") or "homer" + return f"{entity}/emails/test-thread.md" + async def read_capture(self, path): # The simplest re-readable shape for reprocess tests. return self._stored.get(path) @@ -1131,12 +1139,14 @@ async def test_seed_propagates_to_envelope(self): class TestCaptureEmail: - """Email is one more capture source: body → text, subject → title, - Message-ID → `mid:` pointer, filed as an `email`-kind capture through - the same pipeline as URLs and notes. Slice A1 of the email feature.""" + """Email is a capture source that *accumulates*: body → text, subject + → title, but every message folds into one thread file keyed by the + thread root, not a single-shot entry. So it routes through the mirror's + `publish_email_message` (append a section) rather than `publish_capture` + (replace). The classify/tag/envelope tail is shared with URLs/notes.""" @pytest.mark.asyncio - async def test_files_email_as_email_kind_capture(self): + async def test_files_email_through_thread_fold_path(self): mirror = FakeMirror() pipe = _pipeline(mirror=mirror) out = await pipe.capture_email( @@ -1147,10 +1157,37 @@ async def test_files_email_as_email_kind_capture(self): from_addr="schule@example.org", ) assert out.status == "captured" - cap = mirror.captures[0] - assert cap["kind"] == "email" - assert cap["source_uri"] == "mid:abc123@school.example" - assert cap["title_hint"] == "Elternabend am Freitag" + # Routed to the fold path, not the single-shot capture path. + assert mirror.captures == [] + assert len(mirror.emails) == 1 + msg = mirror.emails[0] + # Thread-starting message is its own thread root. + assert msg["thread_uri"] == "mid:abc123@school.example" + assert msg["message_id"] == "" + assert msg["from_addr"] == "schule@example.org" + assert msg["title_hint"] == "Elternabend am Freitag" + assert msg["body_text"] == "Bitte das Formular bis Freitag zurücksenden." + + @pytest.mark.asyncio + async def test_reply_folds_into_same_thread(self): + # Two messages, distinct Message-IDs, same thread root → both land + # in the same thread file (same thread_uri) as separate sections. + mirror = FakeMirror() + pipe = _pipeline(mirror=mirror) + await pipe.capture_email( + subject="Elternabend", body="first message", + message_id="", sender_mxid="@homer:s", + ) + await pipe.capture_email( + subject="Re: Elternabend", body="the reply", + message_id="", + thread_root="", + sender_mxid="@homer:s", + ) + assert len(mirror.emails) == 2 + assert mirror.emails[0]["thread_uri"] == "mid:root@school.example" + assert mirror.emails[1]["thread_uri"] == "mid:root@school.example" + assert mirror.emails[0]["message_id"] != mirror.emails[1]["message_id"] @pytest.mark.asyncio async def test_empty_body_is_empty_outcome(self): @@ -1160,4 +1197,5 @@ async def test_empty_body_is_empty_outcome(self): subject="x", body=" ", message_id="", sender_mxid="@homer:s", ) assert out.status == "empty" + assert mirror.emails == [] assert mirror.captures == [] diff --git a/tests/stacklets/test_extractors.py b/tests/stacklets/test_extractors.py index f077755..4f6c649 100644 --- a/tests/stacklets/test_extractors.py +++ b/tests/stacklets/test_extractors.py @@ -285,32 +285,32 @@ class TestEmailToSource: """`email_to_source` maps fetched email parts into SourceContent. Pure mapping, no I/O — the himalaya container already fetched the - message. The Message-ID becomes an RFC 2392 `mid:` pointer for dedupe - and reprocess.""" + message. The thread root becomes an RFC 2392 `mid:` pointer that keys + the thread file (every reply folds into the same entry).""" - def test_maps_subject_body_and_message_id(self): + def test_maps_subject_body_and_thread_root(self): s = email_to_source( subject="Elternabend am Freitag", body="Bitte Formular zurücksenden.", - message_id="", + thread_root="", ) assert isinstance(s, SourceContent) assert s.text == "Bitte Formular zurücksenden." assert s.title_hint == "Elternabend am Freitag" assert s.source_uri == "mid:abc123@school.example" - def test_strips_angle_brackets_from_message_id(self): - s = email_to_source(subject="x", body="y", message_id=" ") + def test_strips_angle_brackets_from_thread_root(self): + s = email_to_source(subject="x", body="y", thread_root=" ") assert s.source_uri == "mid:id@h" def test_blank_subject_is_none(self): - s = email_to_source(subject=" ", body="y", message_id="") + s = email_to_source(subject=" ", body="y", thread_root="") assert s.title_hint is None - def test_missing_message_id_has_no_source_uri(self): - s = email_to_source(subject="x", body="y", message_id=None) + def test_missing_thread_root_has_no_source_uri(self): + s = email_to_source(subject="x", body="y", thread_root=None) assert s.source_uri is None - s2 = email_to_source(subject="x", body="y", message_id="") + s2 = email_to_source(subject="x", body="y", thread_root="") assert s2.source_uri is None diff --git a/tests/stacklets/test_vault_entry.py b/tests/stacklets/test_vault_entry.py index 421ae2b..95e366e 100644 --- a/tests/stacklets/test_vault_entry.py +++ b/tests/stacklets/test_vault_entry.py @@ -19,6 +19,12 @@ capture_frontmatter, render_document, render_capture, + email_mid_marker, + render_email_message_section, + render_email_thread, + split_frontmatter, + merge_email_frontmatter, + fold_email_message, ) @@ -545,3 +551,184 @@ def test_note_without_body(self): facts=[], ) assert "> [!quote]" not in content # no body → no quote block + + +# ── Email thread folding ─────────────────────────────────────────────── + +class TestRenderEmailMessageSection: + + def test_marker_heading_briefing_and_body(self): + section = render_email_message_section( + message_id="m1@school.example", + from_addr="office@springfield-school.example", + captured_at="2026-06-21", + body="Bitte das Formular zurücksenden.", + summary="School wants the form back.", + facts=["Deadline Friday"], + action_items=[{"action": "Return form", "due": "2026-06-26"}], + ) + # Idempotency marker carries the bare Message-ID. + assert "" in section + # Heading is date — sender. + assert "## 2026-06-21 — office@springfield-school.example" in section + # Per-message briefing callout + action item checkbox. + assert "> [!summary]" in section + assert "- [ ] Return form — 2026-06-26" in section + # Verbatim body in a collapsible quote callout. + assert "> [!quote]- Message" in section + assert "> Bitte das Formular zurücksenden." in section + + def test_no_message_id_omits_marker(self): + section = render_email_message_section( + message_id=None, from_addr="a@b", captured_at="2026-06-21", + body="hi", + ) + assert "" in out + + def test_reply_appends_section_and_unions_frontmatter(self): + first = fold_email_message( + None, + section=self._section("root@h"), message_id="root@h", + new_frontmatter=self._fm(["Homer"], ["school"]), + title="Elternabend", captured_at="2026-06-21", + source_uri="mid:root@h", persons=["Homer"], tags=["school"], + from_path="family/emails/2026/06/x.md", shared_bucket="family", + ) + second = fold_email_message( + first, + section=self._section("reply@h", body="the reply"), + message_id="reply@h", + new_frontmatter=self._fm(["Marge"], ["form"]), + title="Elternabend", captured_at="2026-06-22", + source_uri="mid:root@h", persons=["Marge"], tags=["form"], + from_path="family/emails/2026/06/x.md", shared_bucket="family", + ) + # Both message markers present → one growing file. + assert "" in second + assert "" in second + assert "the reply" in second + # Frontmatter unioned the reply's person and tag. + fm, _ = split_frontmatter(second) + assert fm["persons"] == ["Homer", "Marge"] + assert fm["tags"] == ["school", "form"] + + def test_already_folded_message_is_noop(self): + first = fold_email_message( + None, + section=self._section("root@h"), message_id="root@h", + new_frontmatter=self._fm(["Homer"], ["school"]), + title="Elternabend", captured_at="2026-06-21", + source_uri="mid:root@h", persons=["Homer"], tags=["school"], + from_path="family/emails/2026/06/x.md", shared_bucket="family", + ) + again = fold_email_message( + first, + section=self._section("root@h"), message_id="root@h", + new_frontmatter=self._fm(["Homer"], ["school"]), + title="Elternabend", captured_at="2026-06-21", + source_uri="mid:root@h", persons=["Homer"], tags=["school"], + from_path="family/emails/2026/06/x.md", shared_bucket="family", + ) + assert again is None # idempotent: marker already present + + def test_marker_helper_matches_section(self): + section = self._section("abc@h") + assert email_mid_marker("abc@h") in section From eae94350e95d1d05cff84eaf938a8228bafd9e9c Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 22 Jun 2026 18:05:54 +0200 Subject: [PATCH 09/45] docs: diagram the email thread-fold path; mark threading shipped MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a mermaid flow of fetch → thread_root → fold (first/append/idempotent) to the email design doc, updates the threading section to what landed (per-message append, frontmatter union, mid: marker idempotency), and fixes the stale /mail/ path to /emails/. Section headings use the vault's middot separator, not an em dash. --- docs/design/agent/email-tools.md | 61 ++++++++++++++++++++++------- stacklets/docs/bot/vault_entry.py | 2 +- tests/stacklets/test_vault_entry.py | 6 +-- 3 files changed, 51 insertions(+), 18 deletions(-) diff --git a/docs/design/agent/email-tools.md b/docs/design/agent/email-tools.md index 327b116..51742b3 100644 --- a/docs/design/agent/email-tools.md +++ b/docs/design/agent/email-tools.md @@ -155,8 +155,8 @@ into the container, like other stacklets' CLIs. ▼ CapturePipeline (EXISTING) → classify → mirror to vault → dev.famstack.event │ - ├─ vault entry: /mail/YYYY/MM/-.md (type: email) - │ with `## Action items` already extracted + ├─ vault entry: /emails/YYYY/MM/-.md (type: email) + │ folded by thread; action items extracted per message ├─ tasks rollup (deriver/wiki pattern) → /tasks.md └─ mail bot routes the filing event → the Matrix room whose dev.famstack.capture {kind: mailbox, account, folder} binding matches @@ -243,20 +243,53 @@ An email thread *is* a reply chain, and [adr-010](../adr/adr-010-event-pipeline. already says a filing's source is the whole thread and the vault entry is the *fold* of it. So: -- **Key the entry by the thread root, not the message** — resolved from - `References` / `In-Reply-To` (root Message-ID). Get this right up front; it - changes `capture_email`'s identity model and is painful to retrofit. -- **Each new message updates the same file** — appends the message, re-renders - the conversation chronologically, refreshes a thread-level summary + the - accumulated action items. +```mermaid +flowchart TD + A["MailFetcher.fetch_new(seen)
read-only IMAP, dedup by Message-ID"] --> B[parse_email → ParsedEmail] + B --> C{"thread_root
References[0] / In-Reply-To / own id"} + C --> D["capture_email(thread_root, message_id, from_addr)"] + D --> E["CapturePipeline._publish
classify · tags · envelope (shared with URL/note)"] + E --> F{"kind == email
and email_meta?"} + F -- no --> G[publish_capture: replace single-shot entry] + F -- yes --> H["publish_email_message
path = bucket/emails/YYYY/MM/<slug>-<thread-hash>.md"] + H --> I{thread file exists?} + I -- no --> J["render_email_thread
frontmatter + H1 + meta + first section"] + I -- yes --> K{"<!-- mid:id -->
already in file?"} + K -- yes --> L["no-op (idempotent)"] + K -- no --> M["append dated section
union persons + tags into frontmatter"] + J --> N[("memory.git
one file per thread")] + M --> N + E --> O["dev.famstack.event → bound Matrix room"] +``` + +The fold rules: + +- **Key the entry by the thread root, not the message** — resolved by + `ParsedEmail.thread_root` (`References[0]` → `In-Reply-To` → own Message-ID). + `email_to_source` keys the vault entry's `mid:` URI off the thread root, so + every reply resolves to the same file. (Shipped.) +- **Each new message appends a dated section** — `publish_email_message` reads + the thread file, appends `## YYYY-MM-DD · sender` with the message's own + briefing (summary, facts, action items) and the verbatim body, and writes it + back. Persons and tags **union** into the thread's frontmatter so indexing + spans the whole conversation. (Shipped.) +- **Idempotent by construction** — each section carries an HTML-comment + `` marker; folding a message whose marker is already + present is a no-op. The file itself records which messages it contains, so a + re-run never double-files even before the fetcher's seen-set persists. The + vault is the source of truth (ADR-010), not a side cache. (Shipped.) - **Reproducible**: re-fetch every message in the thread by Message-ID, re-fold. - **Append vs. correct** — both are "reply chains" but mean different things: a - real inbound email *appends* to the conversation; a Matrix reply from a family - member *corrects* the filing (rewrites). The sender / Message-ID distinguishes - them. The thread folder must not treat a human correction as a new email. - -Staging: A2 ships per-message append into the thread file; the polished thread -summary + folded action items can iterate. Thread-keying is day-one. + real inbound email *appends* (routes through `publish_email_message`, carrying + per-message `email_meta`); a Matrix reply from a family member *corrects* the + filing (a rewrite via `reprocess`). The presence of `email_meta` is what + distinguishes them, so a human correction is never folded as a new email. + +Staged status: thread-keying, per-message append, frontmatter union, and the +idempotency marker are **shipped**. Still to iterate: a *thread-level* rolled-up +summary (today each message keeps its own briefing), folding action items into a +single thread checklist, and wiring email `reprocess` to re-fold the whole +thread. ### Summary gate for short mail diff --git a/stacklets/docs/bot/vault_entry.py b/stacklets/docs/bot/vault_entry.py index 5d89540..09481b2 100644 --- a/stacklets/docs/bot/vault_entry.py +++ b/stacklets/docs/bot/vault_entry.py @@ -532,7 +532,7 @@ def render_email_message_section( if message_id: parts.append(email_mid_marker(message_id)) heading_bits = [b for b in (captured_at, from_addr) if b] - parts.append("## " + (" — ".join(heading_bits) or "Message")) + parts.append("## " + (" · ".join(heading_bits) or "Message")) parts.append("") briefing = _briefing_block( diff --git a/tests/stacklets/test_vault_entry.py b/tests/stacklets/test_vault_entry.py index 95e366e..900f75e 100644 --- a/tests/stacklets/test_vault_entry.py +++ b/tests/stacklets/test_vault_entry.py @@ -570,7 +570,7 @@ def test_marker_heading_briefing_and_body(self): # Idempotency marker carries the bare Message-ID. assert "" in section # Heading is date — sender. - assert "## 2026-06-21 — office@springfield-school.example" in section + assert "## 2026-06-21 · office@springfield-school.example" in section # Per-message briefing callout + action item checkbox. assert "> [!summary]" in section assert "- [ ] Return form — 2026-06-26" in section @@ -584,7 +584,7 @@ def test_no_message_id_omits_marker(self): body="hi", ) assert "