From d42e556c1922f1fcab8f1032ce2413bf8215a8ee Mon Sep 17 00:00:00 2001 From: distbit Date: Tue, 6 Jan 2026 08:22:24 +0700 Subject: [PATCH 01/27] chore: checkpoint existing work --- INTERVIEW.md | 26 ---- src/SPEC.md | 334 +++++++++++++++++++++++++++++++++++++++++ src/integrate_notes.py | 15 +- 3 files changed, 345 insertions(+), 30 deletions(-) create mode 100644 src/SPEC.md diff --git a/INTERVIEW.md b/INTERVIEW.md index b9efcef..e69de29 100644 --- a/INTERVIEW.md +++ b/INTERVIEW.md @@ -1,26 +0,0 @@ -# TODO 7: Multiline grouping approach (excluded from LLM body, protected from patches) - -> What exact on-disk syntax do you want for the multiline grouping section? Please provide a concrete before/after example (including where it sits relative to `---` front matter and the `# -- SCRATCHPAD` heading). - -after the front matter, before the scratchpad heading. it should be before any other content in the file after the front matter - -> How should the end of the multiline grouping section be detected (e.g., first blank line, next heading, a closing marker, end-of-file)? Can the grouping text itself contain blank lines? - -use explicit opening and closing marker syntax. if markdown frontmatter satisfies the criteria e.g. support for multiline content, use that. otherwise implement this in the way which makes the most sense / is the most aligned w/ good practice. - -> For legacy documents that still use a single-line `Grouping approach: ...` prefix, should the tool leave that line as-is, or migrate it to the new multiline format when writing the file? If migration is desired, should it happen only when `--grouping` is provided / user is prompted, or always? - -leave as is - -> The current CLI prompt uses `input()` (single line). How do you want multiline grouping input to be entered (e.g., read until a lone `.` line, read until EOF, open $EDITOR, allow literal `\n` escapes, etc.)? - -yes figure out best/most simple but also useable way to support multiline input - -> Should the grouping section be preserved verbatim (whitespace/indentation), or normalized (trim lines, collapse spaces) before inserting into the prompt’s “Maintain the grouping approach: …” line? - -preserved verbatim - -> Do you want the grouping section to be strictly immutable during patch application (i.e., patches only apply to the body after removing the grouping block), or should we also detect and error if a patch’s SEARCH text matches inside the grouping section? - -strictly immutable. it should be as if it didn't exist in the body at all. so if a search block matches it and nothing else, it results in an error > retry. there should not need to be any special handling logic for these cases. it simply isn't part of the document body for the purposes of search/replace or substitute blocks. - diff --git a/src/SPEC.md b/src/SPEC.md new file mode 100644 index 0000000..265fafc --- /dev/null +++ b/src/SPEC.md @@ -0,0 +1,334 @@ +# Zettelkasten Inbox Integration Script — Specification + +## Overview + +A script that automatically integrates new text from an inbox note into the most relevant existing notes in a zettelkasten-style markdown repository, using LLM-guided exploration of the note graph. + +--- + +## Phase 1: Chunking + +### Input + +* Inbox file containing new text to integrate +* Filenames of 15 randomly sampled files (>300 words each, excluding index notes) from the repository + +### Process + +1. Number each paragraph in the inbox +2. Provide LLM with: + + * The numbered paragraphs + * The 15 sampled filenames (for granularity calibration) +3. LLM returns groups of paragraph numbers representing semantically coherent chunks + +### Constraints + +* Max 600 words per chunk (but never split a single paragraph) +* Paragraphs within a chunk need not be contiguous +* Groups should only combine paragraphs that are clearly same topic/chain of thought + +### Rationale + +* LLM chunking ensures semantic coherence; mechanical chunking conflates proximity with relatedness +* Sampled filenames calibrate the LLM to match existing note granularity (filenames alone convey topic scope without token cost) +* Non-contiguous grouping allows related but separated paragraphs to be processed together + +--- + +## Phase 2: Summary Generation (Preprocessing) + +### Cache Invalidation + +* Store `(file_path, content_hash, summary)` tuples +* Regenerate summary when `hash(current_content) != cached_hash` + +### Summary Generation Rules + +**Standard notes:** + +``` +Generate a 75-100 word summary of this note's content. +Focus on: main topics, key claims, what questions it answers. +``` + +**Index notes** (filename contains "index"): + +``` +Generate a summary based on these summaries of linked notes: +[summaries of all notes linked from this file] +Synthesize into 75-100 words describing what this index covers. +``` + +### Rationale + +* Hash-based invalidation is precise—updates exactly when needed +* Index notes contain mostly links; summarizing their linked content is more informative than summarizing the links themselves + +--- + +## Phase 3: Exploration + +### State Model + +Each file can be in one of three states: + +| State | What LLM sees | How it gets there | +| --------------- | ---------------------------------------------- | --------------------------- | +| **Available** | Filename + summary | Linked from a viewed file | +| **Viewed** | Filename + summary + headings + outgoing links | LLM requested to view it | +| **Checked out** | Full content | LLM selected it for editing | + +### Exploration Flow + +``` +1. Initialize: + - Root file is automatically VIEWED (summary + headings + links shown) + - All files linked from root are AVAILABLE (filename + summary shown) + +2. Exploration loop: + a. LLM sees: chunk + all VIEWED files (summary/headings/links) + AVAILABLE files (filename/summary) + b. LLM returns: list of AVAILABLE files to VIEW (up to 4 per round) + c. For each requested file: + - Change state to VIEWED + - Show summary + headings + outgoing links + - Files it links to become AVAILABLE (if not already viewed) + d. Repeat until LLM signals ready OR limits reached + +3. Checkout: + - LLM selects up to 3 VIEWED files to CHECK OUT + - Full content of checked-out files shown + +4. Edit: + - LLM provides find/replace blocks for checked-out files +``` + +### Limits + +* Max 3 exploration rounds +* Up to 4 files may be VIEWED per round (fewer is fine) +* Max 15 files VIEWED total +* Max 3 files CHECKED OUT + +### Context Management + +* Only summaries (not full content) accumulate during exploration +* Full content only loaded at checkout +* Keeps exploration cheap regardless of depth + +### Rationale + +* Three-state model separates cheap browsing from expensive content loading +* AVAILABLE shows summary so LLM can judge relevance; VIEWED adds structure (headings + links) for navigation decisions +* Summaries + headings provide enough signal for navigation decisions +* Root file treated identically to others; may itself be edited or contain no links + +--- + +## Phase 4: Editing + +### Edit Format + +```json +{ + "edits": [ + { + "file": "filename.md", + "find": "exact text to locate", + "replace": "replacement text", + "is_duplicate": false + }, + { + "file": "other.md", + "find": "text that already covers this", + "is_duplicate": true + } + ] +} +``` + +### Edit Types + +**Standard edit:** `find` + `replace` provided, content is modified + +**Insertion:** `find` contains anchor text, `replace` contains anchor + new content + +```json +{ + "find": "- Link B", + "replace": "- Link B\n- Link C" +} +``` + +**Duplicate marker:** `is_duplicate: true`, only `find` required + +* `find` contains existing text that already covers the chunk content +* No replacement made; serves as visibility into why content wasn't added + +### Validation + +1. For each edit, search for `find` text in specified file using a whitespace-normalized match (treat runs of spaces/tabs/newlines as equivalent, and ignore trivial leading/trailing whitespace differences) to increase match reliability +2. Must match exactly once (zero matches = error, multiple matches = error) +3. On validation failure: return error to LLM, request correction within same conversation + +### Scope + +* Edits can target any CHECKED OUT file +* This includes the root file and index notes + +### Rationale + +* Find/replace is simple and unambiguous; insertion is just a usage pattern, not a separate operation +* Single-match requirement prevents ambiguous edits +* Duplicate flag provides audit trail without cluttering output with identical find/replace pairs +* In-conversation correction leverages existing context rather than restarting + +--- + +## Tool Schema + +### Exploration Tools + +```typescript +// Request to view files (see headings + links in addition to summary) +interface ViewFilesRequest { + action: "view"; + files: string[]; // up to 4, must be AVAILABLE +} + +// Signal ready to check out files for editing +interface CheckoutRequest { + action: "checkout"; + files: string[]; // max 3, must be VIEWED +} +``` + +### Edit Tools + +```typescript +interface Edit { + file: string; + find: string; + replace?: string; // omit if is_duplicate + is_duplicate: boolean; +} + +interface EditRequest { + action: "edit"; + edits: Edit[]; +} +``` + +--- + +## File Annotation Format + +When displaying a VIEWED file: + +```markdown +## [filename.md] + +**Summary:** [75-100 word summary] + +**Headings:** +- # Main Title +- ## Section One +- ## Section Two +- ### Subsection + +**Links to:** +- [[other-note.md]] — [summary of other-note] +- [[another.md]] — [summary of another] +``` + +When displaying an AVAILABLE file: + +```markdown +- [[filename.md]] — [75-100 word summary] +``` + +--- + +## Execution Flow Summary + +``` +┌─────────────────────────────────────────────────────────┐ +│ PREPROCESSING (run periodically) │ +│ - Update stale summaries (hash-based invalidation) │ +│ - Index notes: summarize from linked summaries │ +└─────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────┐ +│ CHUNKING │ +│ - Show LLM: inbox paragraphs + 15 sample filenames │ +│ - LLM returns: paragraph groupings (max 600w each) │ +└─────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────┐ +│ FOR EACH CHUNK: │ +│ │ +│ ┌───────────────────────────────────────────────────┐ │ +│ │ EXPLORE (max 3 rounds, up to 4 files per round, │ │ +│ │ max 15 files viewed total) │ │ +│ │ - AVAILABLE: see filename + summary │ │ +│ │ - VIEWED: see summary + headings + links │ │ +│ │ - Request more files or signal ready │ │ +│ └───────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌───────────────────────────────────────────────────┐ │ +│ │ CHECKOUT (max 3 files) │ │ +│ │ - Load full content of selected files │ │ +│ └───────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌───────────────────────────────────────────────────┐ │ +│ │ EDIT │ │ +│ │ - LLM provides find/replace blocks │ │ +│ │ - Validate single-match constraint │ │ +│ │ - Apply edits or request correction │ │ +│ └───────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────┘ +``` + +--- + +## Configuration + +```yaml +# Limits +max_exploration_rounds: 3 +max_files_viewed_per_round: 4 +max_files_viewed_total: 15 +max_files_checked_out: 3 +max_chunk_words: 600 +granularity_sample_size: 15 +granularity_sample_min_words: 300 + +# Paths +root_file: "index.md" +inbox_file: "inbox.md" +notes_directory: "./notes" +summary_cache: "./.summary_cache.json" + +# Summary +summary_target_words: 75-100 +index_filename_pattern: "index" +``` + +--- + +## Out of Scope (Deliberate Simplifications) + +| Feature | Reason excluded | +| --------------------------------- | --------------------------------------------------------------- | +| Create new note | Adds complexity; can be added later | +| Explicit insert_after operation | Find/replace pattern sufficient | +| Summary update debouncing | Premature optimization | +| Pre-routing with all summaries | Doesn't scale; exploration achieves same goal | +| Confidence ratings / review queue | Adds friction; start simple | +| Multiple root files / fallbacks | Unnecessary if root is maintained | +| Heading-level routing | Single dimension (files) is simpler than two (files + sections) | \ No newline at end of file diff --git a/src/integrate_notes.py b/src/integrate_notes.py index 41f758f..398bd9f 100644 --- a/src/integrate_notes.py +++ b/src/integrate_notes.py @@ -201,7 +201,10 @@ def extract_grouping_section(body: str) -> tuple[GroupingSection | None, str]: while grouping_index < len(lines) and not lines[grouping_index].strip(): grouping_index += 1 - if grouping_index < len(lines) and lines[grouping_index].strip() == GROUPING_BLOCK_START: + if ( + grouping_index < len(lines) + and lines[grouping_index].strip() == GROUPING_BLOCK_START + ): end_index = grouping_index + 1 while end_index < len(lines) and lines[end_index].strip() != GROUPING_BLOCK_END: end_index += 1 @@ -242,7 +245,9 @@ def _format_grouping_block(grouping_text: str) -> str: def render_grouping_section( - grouping_text: str, existing_section: GroupingSection | None, preserve_existing: bool + grouping_text: str, + existing_section: GroupingSection | None, + preserve_existing: bool, ) -> str: if not grouping_text.strip(): raise ValueError("Grouping approach cannot be empty.") @@ -287,7 +292,7 @@ def prompt_for_grouping() -> str: f"{GROUPING_PREFIX} at the top of the document.\n" "Enter multiline text and finish with a single line containing only a '.'.\n" "Examples:\n" - "- Grouping approach: Group points according to what problem each idea/proposal/mechanism/concept addresses/are trying to solve, which you will need to figure out yourself based on context. Do not combine multiple goals/problems into one group. Keep goals/problems specific. Ensure groups are mutually exclusive and collectively exhaustive. Avoid overlap between group's goals/problems. sub-headings should be per-mechanism/per-solution i.e. according to which \"idea\"/solution each point relates to.\n" + '- Grouping approach: Group points according to what problem each idea/proposal/mechanism/concept addresses/are trying to solve, which you will need to figure out yourself based on context. Do not combine multiple goals/problems into one group. Keep goals/problems specific. Ensure groups are mutually exclusive and collectively exhaustive. Avoid overlap between group\'s goals/problems. sub-headings should be per-mechanism/per-solution i.e. according to which "idea"/solution each point relates to.\n' "- Group points according to what you think the most useful/interesting/relevant groupings are. Ensure similar, related and contradictory points are adjacent.\n" "Your input:\n" ) @@ -1391,7 +1396,9 @@ def integrate_notes( source_body, source_scratchpad = split_document_sections(source_content) grouping_section, working_body = extract_grouping_section(source_body) - resolved_grouping = grouping or (grouping_section.text if grouping_section else None) + resolved_grouping = grouping or ( + grouping_section.text if grouping_section else None + ) if not resolved_grouping: resolved_grouping = prompt_for_grouping() logger.info("Recorded new grouping approach from user input.") From f499460d201b8a4d8de1fd8f436c7368b25785ec Mon Sep 17 00:00:00 2001 From: distbit Date: Tue, 6 Jan 2026 08:46:12 +0700 Subject: [PATCH 02/27] docs: capture spec interview answers --- INTERVIEW.md | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/INTERVIEW.md b/INTERVIEW.md index e69de29..f1b27ac 100644 --- a/INTERVIEW.md +++ b/INTERVIEW.md @@ -0,0 +1,92 @@ +# SPEC.md Implementation + +## Scope & entry point +> Should the SPEC flow fully replace the current scratchpad-based integration in `src/integrate_notes.py`, or should it live as a new CLI/module (e.g., `src/zet_integrator.py`) with its own entry point? + +the inbox referred to in the spec i.e. where the text is coming from should still be the scratchpad of the file specified in the cli, the same way it is currently. the "root" note mentioned in the spec is just whichever file is passed to integrate_notes.py i.e. the file which we read the scratchpad of. + +> If it replaces the current flow, should we remove the existing scratchpad/grouping CLI flags and related logic entirely (per “no backward compatibility”), or keep any pieces (which ones)? + +ok actually so what i want to do is keep the old version of integrate_notes.py untouched, but implement the SPEC.md in a new py file, and in this file, remove all grouping instructions related logic. but keep scratchpad stuff, as that is where the notes which are being chunked/integrated are coming from. + +## Notes repository & config +> What are the concrete paths for `notes_directory`, `root_file`, and `inbox_file` in your environment (relative to repo or absolute), and should they be set via `config.json`, CLI flags, or environment variables (and which should take precedence)? + +inbox file is just the scratchpad of root file, and root file is just the file provided via the cli, and notes directory is just whatever directory root file is in. inbox file is not a separate file, despite what the spec says. + +> For `index_filename_pattern`, is the simple substring match in the spec (“index”) correct for your notes, or do you want a stricter rule (e.g., suffix/prefix/regex)? + +if it ends in index.md, it is an index file. + +> Should we treat non-note markdown files in the notes directory (templates, archives, etc.) as excluded by default? If yes, how do you want them identified (folder name, filename pattern, front matter flag)? + +files are only "included" if they linked to, directly or indirectly (through another file) from the root file. only markdown files can be linked to. the fact that you ask me this seems to indicate you have some confusion though, as i don't see a need to exclude files just because they are in the dir. we don't scan all files in the dir. we only look at linked files and derive their path from the link text, .md and the root file dir (noting that links are case insensitive though) + +## Chunking +> When numbering paragraphs in the inbox, do you want paragraphs split strictly on blank lines (current `normalize_paragraphs` behavior), or should we treat other separators (e.g., horizontal rules) as paragraph boundaries too? + +just blank lines + +> For non-contiguous chunk groups returned by the model, should we preserve original paragraph order when assembling each chunk, or should we follow the order returned by the model? + +the latter. + +> For the “15 randomly sampled filenames (>300 words)” calibration step, should the sample be deterministic (seeded) for reproducibility, and should word counts ignore front matter/code blocks? + +doesn't need to be seeded. doesn't need to ignore front matter. note that these must only be files which are linked to (recursively/indirectly or directly) from the root note. not just any files in the dir. + +## Summaries & cache +> For summary cache storage, should we use the spec’s `.summary_cache.json` at repo root, or do you prefer a different location/format (and should it be tracked in git or ignored)? + +no put it outside the directory to avoid creating a mess. + +> For index-note summaries (summaries of linked notes), should we include links that resolve outside `notes_directory`, or only within it? How should broken/missing links be handled (error vs. skip)? + +skip links for which no notes file exists. links are NOT capable of resolving outside of the notes directory, as the link text just specifies the file name and the directory is always implicitly notes directory. + +> Should summary generation happen as a separate command (preprocessing), or on-demand during integration if a needed summary is stale/missing? + +on-demand. but make sure it is maximally parallelised, to avoid needing to wait a long time. + +## Markdown parsing & links +> Which link syntaxes should count as “outgoing links” for exploration: `[[wikilink]]`, `[text](file.md)`, bare `file.md`, or something else? Any special handling for anchors like `[[note#section]]`? + +only wikilinks. if it has a #section in the wikilink, just ignore the section. too complex. + +> For headings extraction, should we only parse ATX `#` headings (ignore Setext), and should headings inside code blocks be ignored? + +ignore setext. do not ignore headings inside code blocks. too complex + +## LLM interaction & tools +> Do you want true tool calling via OpenAI Responses API tools (structured `action: view/checkout/edit`), or is strict JSON schema parsing of a normal response sufficient as long as the schema matches the spec? + +yes use tools. + +> Which model and reasoning level should we use for chunking, summaries, exploration, and edit phases? Should they all be the same (currently `gpt-5.2`/`medium`) or different per phase? + +yes use medium for all. + +> When an edit validation fails (0 or >1 matches), should we send the model just the failure details, or also include the specific matching candidates/snippets from the file to help it correct? + +just failure details. + +## Editing & validation +> The spec’s whitespace-normalized matching treats runs of whitespace as equivalent. Should we still allow the exact/trimmed match fallbacks currently used in `_locate_search_text`, or switch strictly to the whitespace-normalized approach only? + +both (but simplify if possible). be quite permissive here. we like to avoid unnecessary matching errors. + +> For `is_duplicate: true` edits, do you want them surfaced in output/logs (and how), or just kept internally for audit? + +preserve the existing verification logic, but update it to support these new edit request formats. + +## Logging & runtime behavior +> Where should logs live (the spec doesn’t say, but project rules say “move logs outside src/”)? Is `./logs/integrate_notes.log` acceptable, or do you want a different path/rotation policy? + +yes that is ok. + +> Should the integration run be single-threaded for determinism, or is limited concurrency acceptable for summary generation/exploration? + +concurrency is desired for any api calls. + + +also pls split up the code, including the existing logic, across multiple semantically-meaningful files each focusing on a different area, as the current file is wayy too long. From 2213842c5ffba940a926ce6cde488df6525797b0 Mon Sep 17 00:00:00 2001 From: distbit Date: Tue, 6 Jan 2026 08:48:01 +0700 Subject: [PATCH 03/27] docs: capture spec follow-up answers --- INTERVIEW.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/INTERVIEW.md b/INTERVIEW.md index f1b27ac..f4fdc7c 100644 --- a/INTERVIEW.md +++ b/INTERVIEW.md @@ -90,3 +90,12 @@ concurrency is desired for any api calls. also pls split up the code, including the existing logic, across multiple semantically-meaningful files each focusing on a different area, as the current file is wayy too long. + +## Follow-ups +> You asked to keep the old `integrate_notes.py` untouched but also to split the existing logic into multiple files. Which should take priority: (a) leave `integrate_notes.py` as-is and only add new modules for the SPEC flow, or (b) refactor the existing logic into multiple files while keeping behavior unchanged? + +keep the old one. but duplicate it and refactor + implement new logic in this duplicate version. + +> For the summary cache “outside the directory,” do you want it outside the notes directory but still inside the repo (e.g., `./.summary_cache.json`), or in a user-level cache dir (e.g., `~/.cache/integrate_notes/summary_cache.json`)? If user-level, which exact path should we use on Fedora? + +user level. up to you re: path From 22c3b5061b6510d1dcdcd5eb61e2810ef6467460 Mon Sep 17 00:00:00 2001 From: distbit Date: Tue, 6 Jan 2026 09:14:44 +0700 Subject: [PATCH 04/27] feat: add spec-based integration flow --- src/integrate_notes_spec.py | 287 +++++++++++++++++++++++++++ src/spec_chunking.py | 120 +++++++++++ src/spec_config.py | 82 ++++++++ src/spec_editing.py | 383 ++++++++++++++++++++++++++++++++++++ src/spec_exploration.py | 282 ++++++++++++++++++++++++++ src/spec_llm.py | 111 +++++++++++ src/spec_logging.py | 24 +++ src/spec_markdown.py | 74 +++++++ src/spec_notes.py | 87 ++++++++ src/spec_summary.py | 204 +++++++++++++++++++ src/spec_verification.py | 327 ++++++++++++++++++++++++++++++ 11 files changed, 1981 insertions(+) create mode 100644 src/integrate_notes_spec.py create mode 100644 src/spec_chunking.py create mode 100644 src/spec_config.py create mode 100644 src/spec_editing.py create mode 100644 src/spec_exploration.py create mode 100644 src/spec_llm.py create mode 100644 src/spec_logging.py create mode 100644 src/spec_markdown.py create mode 100644 src/spec_notes.py create mode 100644 src/spec_summary.py create mode 100644 src/spec_verification.py diff --git a/src/integrate_notes_spec.py b/src/integrate_notes_spec.py new file mode 100644 index 0000000..0bbebc7 --- /dev/null +++ b/src/integrate_notes_spec.py @@ -0,0 +1,287 @@ +from __future__ import annotations + +import argparse +import random +import sys +from pathlib import Path +from time import perf_counter + +from loguru import logger + +from spec_chunking import request_chunk_groups +from spec_config import ( + SCRATCHPAD_HEADING, + SpecConfig, + default_log_path, + load_config, + repo_root, +) +from spec_editing import request_and_apply_edits +from spec_exploration import explore_until_checkout +from spec_llm import create_openai_client +from spec_logging import configure_logging +from spec_markdown import ( + build_document, + format_duration, + normalize_paragraphs, + split_document_sections, +) +from spec_notes import NoteRepository +from spec_summary import SummaryService +from spec_verification import VerificationManager, build_verification_prompt + + +def parse_arguments() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Integrate scratchpad notes into a markdown repository (SPEC flow)." + ) + parser.add_argument( + "--source", required=False, help="Path to the root markdown document." + ) + parser.add_argument( + "--disable-verification", + action="store_true", + help="Disable verification prompts and background verification checks.", + ) + return parser.parse_args() + + +def resolve_source_path(provided_path: str | None) -> Path: + if provided_path: + path = Path(provided_path).expanduser().resolve() + else: + user_input = input("Enter path to the root markdown document: ").strip() + if not user_input: + raise ValueError("Document path is required to proceed.") + path = Path(user_input).expanduser().resolve() + if not path.exists(): + raise FileNotFoundError(f"Source document not found at {path}.") + if not path.is_file(): + raise ValueError(f"Source path {path} is not a file.") + return path + + +def _select_sample_filenames( + repo: NoteRepository, reachable: list[Path], config: SpecConfig +) -> list[str]: + candidates = [] + for path in reachable: + if repo.is_index_note(path, config.index_filename_suffix): + continue + if repo.get_word_count(path) < config.granularity_sample_min_words: + continue + candidates.append(path.name) + + if not candidates: + return [] + + if len(candidates) <= config.granularity_sample_size: + return candidates + + return random.sample(candidates, config.granularity_sample_size) + + +def _ensure_scratchpad_matches( + source_path: Path, expected_paragraphs: list[str] +) -> tuple[str, list[str]]: + content = source_path.read_text(encoding="utf-8") + body, scratchpad = split_document_sections(content, SCRATCHPAD_HEADING) + paragraphs = normalize_paragraphs(scratchpad) + if paragraphs != expected_paragraphs: + raise RuntimeError( + "Scratchpad changed while integration was running; aborting to avoid data loss." + ) + return body, paragraphs + + +def _write_updated_files( + source_path: Path, + root_body: str, + remaining_paragraphs: list[str], + updated_files: dict[Path, str], + repo: NoteRepository, + summaries: SummaryService, +) -> None: + for path, content in updated_files.items(): + if path == source_path: + root_body = content + else: + path.write_text(content, encoding="utf-8") + repo.invalidate_content(path) + summaries.invalidate(path) + + document = build_document(root_body, SCRATCHPAD_HEADING, remaining_paragraphs) + source_path.write_text(document, encoding="utf-8") + repo.set_root_body(root_body) + + +def integrate_notes_spec(source_path: Path, disable_verification: bool) -> Path: + config = load_config(repo_root() / "config.json") + source_content = source_path.read_text(encoding="utf-8") + source_body, source_scratchpad = split_document_sections( + source_content, SCRATCHPAD_HEADING + ) + scratchpad_paragraphs = normalize_paragraphs(source_scratchpad) + + repo = NoteRepository(source_path, source_body, source_path.parent) + client = create_openai_client() + summaries = SummaryService(repo, client, config) + verification_manager = ( + None if disable_verification else VerificationManager(client, source_path) + ) + + try: + if not scratchpad_paragraphs: + logger.info( + "No scratchpad notes to integrate; ensuring scratchpad heading remains present." + ) + source_path.write_text( + build_document(source_body, SCRATCHPAD_HEADING, []), + encoding="utf-8", + ) + return source_path + + reachable = repo.iter_reachable_paths() + sample_filenames = _select_sample_filenames(repo, reachable, config) + chunk_groups = request_chunk_groups( + client, scratchpad_paragraphs, sample_filenames, config + ) + + remaining_indices = set(range(1, len(scratchpad_paragraphs) + 1)) + total_chunks = len(chunk_groups) + chunks_completed = 0 + integration_start = perf_counter() + current_body = source_body + + for group in chunk_groups: + if any(index not in remaining_indices for index in group): + raise RuntimeError( + "Chunk references paragraphs that were already integrated; aborting." + ) + chunk_paragraphs = [scratchpad_paragraphs[index - 1] for index in group] + chunk_text = "\n\n".join(chunk_paragraphs) + + expected_remaining = [ + scratchpad_paragraphs[index - 1] + for index in sorted(remaining_indices) + ] + file_body, _ = _ensure_scratchpad_matches( + source_path, expected_remaining + ) + if file_body != current_body: + raise RuntimeError( + "Root document body changed while integration was running; aborting." + ) + + repo.set_root_body(current_body) + reachable = repo.iter_reachable_paths() + summary_map = summaries.get_summaries(reachable) + + root_summary = summary_map[source_path] + root_headings = repo.get_headings(source_path) + root_links = repo.get_links(source_path) + root_link_summaries = [ + (path, summary_map[path]) + for path in root_links + if path in summary_map + ] + + chunk_label = f"chunk {chunks_completed + 1}/{total_chunks}" + checkout_paths = explore_until_checkout( + client, + chunk_text, + source_path, + root_summary, + root_headings, + root_links, + root_link_summaries, + summary_map, + repo, + config, + ) + + checked_out_contents = { + path: repo.get_note_content(path) for path in checkout_paths + } + edit_application = request_and_apply_edits( + client, + chunk_text, + checked_out_contents, + checkout_paths, + chunk_label, + ) + + for path, content in edit_application.updated_contents.items(): + if path == source_path: + current_body = content + for path in edit_application.updated_contents: + if path != source_path: + repo.invalidate_content(path) + + for index in group: + remaining_indices.remove(index) + remaining_paragraphs = [ + scratchpad_paragraphs[index - 1] + for index in sorted(remaining_indices) + ] + + _write_updated_files( + source_path, + current_body, + remaining_paragraphs, + edit_application.updated_contents, + repo, + summaries, + ) + + if verification_manager is not None: + verification_prompt = build_verification_prompt( + chunk_text, + edit_application.patch_replacements, + edit_application.duplicate_texts, + ) + verification_manager.enqueue_prompt( + verification_prompt, + chunk_label, + chunks_completed, + total_chunks, + ) + + chunks_completed += 1 + remaining_chunks = total_chunks - chunks_completed + if remaining_chunks > 0: + elapsed_seconds = perf_counter() - integration_start + average_duration = elapsed_seconds / chunks_completed + estimated_seconds_remaining = average_duration * remaining_chunks + logger.info( + f"Estimated time remaining: {format_duration(estimated_seconds_remaining)}" + f" for {remaining_chunks} remaining chunk(s)." + ) + + logger.info("All scratchpad notes integrated; scratchpad section cleared.") + return source_path + finally: + summaries.shutdown() + if verification_manager is not None: + verification_manager.shutdown() + + +def main() -> None: + configure_logging(default_log_path()) + try: + args = parse_arguments() + source_path = resolve_source_path(args.source) + integrated_path = integrate_notes_spec( + source_path, + args.disable_verification, + ) + logger.info( + f"Integration completed. Updated document available at {integrated_path}." + ) + except Exception as error: + logger.exception(f"Integration failed: {error}") + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/src/spec_chunking.py b/src/spec_chunking.py new file mode 100644 index 0000000..7d0a972 --- /dev/null +++ b/src/spec_chunking.py @@ -0,0 +1,120 @@ +from __future__ import annotations + +import json +from typing import List + +from loguru import logger + +from spec_config import MAX_CHUNKING_ATTEMPTS, SpecConfig +from spec_llm import request_text +from spec_markdown import count_words + + +def build_chunking_prompt( + numbered_paragraphs: List[str], sample_filenames: List[str], config: SpecConfig +) -> str: + paragraphs_block = "\n".join(numbered_paragraphs) + samples_block = "\n".join(f"- {name}" for name in sample_filenames) + + instructions = ( + "Group the numbered paragraphs into semantically coherent chunks. " + "Paragraphs in a chunk need not be contiguous. " + "Do not split a paragraph. " + f"Each chunk must be at most {config.max_chunk_words} words. " + "Return JSON only in the form: {\"groups\": [[1,2],[3]]}. " + "Include every paragraph number exactly once. " + "The order of groups should reflect the order you want them processed; do not sort." + ) + + return ( + "\n" + f"{instructions}\n" + "\n\n" + "\n" + f"{paragraphs_block}\n" + "\n\n" + "\n" + f"{samples_block}\n" + "" + ) + + +def _parse_group_payload(payload: str, total_paragraphs: int) -> List[List[int]]: + data = json.loads(payload) + if not isinstance(data, dict) or "groups" not in data: + raise ValueError("Chunking response must be a JSON object with a 'groups' key.") + groups = data["groups"] + if not isinstance(groups, list) or not groups: + raise ValueError("Chunking response 'groups' must be a non-empty list.") + + seen: set[int] = set() + parsed_groups: List[List[int]] = [] + + for group in groups: + if not isinstance(group, list) or not group: + raise ValueError("Each chunk group must be a non-empty list of integers.") + parsed_group: List[int] = [] + for value in group: + if not isinstance(value, int): + raise ValueError("Chunk group entries must be integers.") + if value < 1 or value > total_paragraphs: + raise ValueError( + f"Paragraph number {value} is out of range 1..{total_paragraphs}." + ) + if value in seen: + raise ValueError(f"Paragraph number {value} appears in multiple groups.") + seen.add(value) + parsed_group.append(value) + parsed_groups.append(parsed_group) + + if len(seen) != total_paragraphs: + missing = [str(i) for i in range(1, total_paragraphs + 1) if i not in seen] + raise ValueError(f"Chunking response missing paragraphs: {', '.join(missing)}") + + return parsed_groups + + +def request_chunk_groups( + client, + paragraphs: List[str], + sample_filenames: List[str], + config: SpecConfig, +) -> List[List[int]]: + numbered_paragraphs = [f"{index + 1}) {text}" for index, text in enumerate(paragraphs)] + feedback: str | None = None + + for attempt in range(1, MAX_CHUNKING_ATTEMPTS + 1): + prompt = build_chunking_prompt(numbered_paragraphs, sample_filenames, config) + if feedback: + prompt += ( + "\n\n\n" + f"{feedback}\n" + "" + ) + response_text = request_text(client, prompt, f"chunking attempt {attempt}") + try: + groups = _parse_group_payload(response_text, len(paragraphs)) + except Exception as error: # noqa: BLE001 + feedback = f"Parsing error: {error}" + logger.warning(f"Chunking response invalid on attempt {attempt}: {error}") + continue + + invalid_group = None + for group in groups: + words = sum(count_words(paragraphs[index - 1]) for index in group) + if words > config.max_chunk_words: + invalid_group = (group, words) + break + if invalid_group: + group, words = invalid_group + feedback = ( + f"Chunk {group} has {words} words, exceeding max {config.max_chunk_words}." + ) + logger.warning( + f"Chunking response exceeded word limit on attempt {attempt}: {feedback}" + ) + continue + + return groups + + raise RuntimeError("Unable to obtain valid chunk grouping from the model.") diff --git a/src/spec_config.py b/src/spec_config.py new file mode 100644 index 0000000..b9af278 --- /dev/null +++ b/src/spec_config.py @@ -0,0 +1,82 @@ +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +SCRATCHPAD_HEADING = "# -- SCRATCHPAD" +ENV_API_KEY = "OPENAI_API_KEY" + +DEFAULT_MODEL = "gpt-5.2" +DEFAULT_REASONING = {"effort": "medium"} + +DEFAULT_MAX_RETRIES = 3 +RETRY_INITIAL_DELAY_SECONDS = 2.0 +RETRY_BACKOFF_FACTOR = 2.0 + +MAX_PATCH_ATTEMPTS = 3 +MAX_TOOL_ATTEMPTS = 3 +MAX_CHUNKING_ATTEMPTS = 3 + +MAX_CONCURRENT_VERIFICATIONS = 4 + +LOG_FILE_ROTATION_BYTES = 2 * 1024 * 1024 + + +@dataclass(frozen=True) +class SpecConfig: + max_exploration_rounds: int = 3 + max_files_viewed_per_round: int = 4 + max_files_viewed_total: int = 15 + max_files_checked_out: int = 3 + max_chunk_words: int = 600 + granularity_sample_size: int = 15 + granularity_sample_min_words: int = 300 + summary_target_words_min: int = 75 + summary_target_words_max: int = 100 + index_filename_suffix: str = "index.md" + + +def repo_root() -> Path: + return Path(__file__).resolve().parent.parent + + +def default_log_path() -> Path: + return repo_root() / "logs" / "integrate_notes.log" + + +def default_pending_prompts_path() -> Path: + return repo_root() / "logs" / "pending_verification_prompts.json" + + +def default_summary_cache_path() -> Path: + return Path.home() / ".cache" / "integrate_notes" / "summary_cache.json" + + +def load_config(config_path: Path) -> SpecConfig: + if not config_path.exists(): + return SpecConfig() + + raw = config_path.read_text(encoding="utf-8") + if not raw.strip(): + return SpecConfig() + + data = json.loads(raw) + if not isinstance(data, dict): + raise ValueError("config.json must contain a JSON object.") + + defaults = SpecConfig() + overrides: dict[str, Any] = {} + for field_name in defaults.__dataclass_fields__: + if field_name not in data: + continue + value = data[field_name] + expected_value = getattr(defaults, field_name) + if not isinstance(value, type(expected_value)): + raise ValueError( + f"config.json field '{field_name}' must be {type(expected_value).__name__}." + ) + overrides[field_name] = value + + return SpecConfig(**{**defaults.__dict__, **overrides}) diff --git a/src/spec_editing.py b/src/spec_editing.py new file mode 100644 index 0000000..7274323 --- /dev/null +++ b/src/spec_editing.py @@ -0,0 +1,383 @@ +from __future__ import annotations + +import re +from dataclasses import dataclass +from pathlib import Path +from typing import Dict, Iterable, List + +from loguru import logger + +from spec_config import MAX_PATCH_ATTEMPTS +from spec_llm import parse_tool_call_arguments, request_tool_call + + +EDIT_TOOL_SCHEMA = { + "type": "function", + "name": "edit_notes", + "description": "Provide find/replace edits for checked-out files.", + "strict": True, + "parameters": { + "type": "object", + "properties": { + "action": {"type": "string", "enum": ["edit"]}, + "edits": { + "type": "array", + "items": { + "type": "object", + "properties": { + "file": {"type": "string"}, + "find": {"type": "string"}, + "replace": {"type": "string"}, + "is_duplicate": {"type": "boolean"}, + }, + "required": ["file", "find", "is_duplicate"], + }, + }, + }, + "required": ["action", "edits"], + }, +} + + +@dataclass(frozen=True) +class EditInstruction: + file_path: Path + find_text: str + replace_text: str | None + is_duplicate: bool + + +@dataclass(frozen=True) +class EditFailure: + index: int + file_path: Path + find_text: str + reason: str + + +@dataclass(frozen=True) +class EditApplication: + updated_contents: Dict[Path, str] + patch_replacements: List[str] + duplicate_texts: List[str] + + +class EditParseError(RuntimeError): + pass + + +def build_edit_prompt( + chunk_text: str, + checked_out_contents: Dict[Path, str], + failed_edits: List[EditFailure] | None = None, + failed_formatting: str | None = None, + previous_response: str | None = None, +) -> str: + file_sections = [] + for path, content in checked_out_contents.items(): + file_sections.append(f"## [{path.name}]\n\n{content}") + + instructions = ( + "You are integrating the notes chunk into the checked-out files. " + "Return only a tool call to edit_notes with edits targeting the listed files. " + "Use is_duplicate=true only when the notes are already fully covered by existing text. " + "For edits, 'find' must be a single contiguous span copied from the file content. " + "For insertions, include the anchor text in both find and replace. " + "Do not include any commentary or additional text." + ) + + prompt = ( + "\n" + f"{instructions}\n" + "\n\n" + "\n" + f"{chunk_text}\n" + "\n\n" + "\n" + f"{'\n\n'.join(file_sections)}\n" + "" + ) + + if failed_formatting or failed_edits: + feedback_lines: List[str] = [] + if failed_formatting: + feedback_lines.append( + "The previous response could not be parsed. Fix the issues below and re-emit a valid tool call." + ) + feedback_lines.append(f"Error: {failed_formatting}") + if failed_edits: + feedback_lines.append( + "The previous edits failed to match the current file contents. Adjust only the failing edits." + ) + for failure in failed_edits: + feedback_lines.append( + f"Edit {failure.index} ({failure.file_path.name}) find text must match exactly once." + ) + feedback_lines.append(failure.find_text) + feedback_lines.append(f"Reason: {failure.reason}") + prompt += ( + "\n\n\n" + + "\n\n".join(feedback_lines) + + "\n" + ) + + if previous_response: + prompt += ( + "\n\n\n" + + previous_response + + "\n" + ) + + return prompt + + +def parse_edit_instructions( + payload: dict, + checked_out_paths: Iterable[Path], +) -> List[EditInstruction]: + action = payload.get("action") + if action != "edit": + raise EditParseError("Edit tool payload must include action='edit'.") + + edits = payload.get("edits") + if not isinstance(edits, list) or not edits: + raise EditParseError("Edit tool payload must include a non-empty edits list.") + + checked_out_map = {path.name.lower(): path for path in checked_out_paths} + instructions: List[EditInstruction] = [] + + for edit in edits: + if not isinstance(edit, dict): + raise EditParseError("Each edit must be an object.") + file_name = edit.get("file") + if not isinstance(file_name, str) or not file_name.strip(): + raise EditParseError("Each edit must include a non-empty file name.") + path = checked_out_map.get(file_name.strip().lower()) + if path is None: + raise EditParseError( + f"Edit file '{file_name}' is not in the checked-out file list." + ) + find_text = edit.get("find") + if not isinstance(find_text, str) or not find_text.strip(): + raise EditParseError("Each edit must include non-empty find text.") + is_duplicate = edit.get("is_duplicate") + if not isinstance(is_duplicate, bool): + raise EditParseError("Each edit must include a boolean is_duplicate flag.") + replace_text = edit.get("replace") + if is_duplicate: + replace_text = None + else: + if not isinstance(replace_text, str): + raise EditParseError( + "Non-duplicate edits must include a string replace value." + ) + instructions.append( + EditInstruction( + file_path=path, + find_text=find_text, + replace_text=replace_text, + is_duplicate=is_duplicate, + ) + ) + + return instructions + + +def _normalize_line_endings(text: str) -> str: + return text.replace("\r\n", "\n").replace("\r", "\n") + + +def _build_whitespace_pattern(text: str, allow_zero: bool) -> re.Pattern[str]: + if not text: + raise ValueError("Cannot build whitespace pattern for empty text.") + + pieces: List[str] = [] + whitespace_token = r"\s*" if allow_zero else r"\s+" + in_whitespace = False + + for char in text: + if char.isspace(): + if not in_whitespace: + pieces.append(whitespace_token) + in_whitespace = True + else: + pieces.append(re.escape(char)) + in_whitespace = False + + pattern = "".join(pieces) + if not pattern: + pattern = whitespace_token + return re.compile(pattern, flags=re.MULTILINE) + + +def _locate_search_text(body: str, search_text: str) -> tuple[int | None, int | None, str]: + attempted_descriptions: List[str] = [] + + index = body.find(search_text) + attempted_descriptions.append("exact match") + if index != -1: + next_index = body.find(search_text, index + len(search_text)) + if next_index != -1: + reason = ( + "SEARCH text matched multiple locations using exact match; " + "increase SEARCH text length to match a longer, more specific span." + ) + return None, None, reason + return index, index + len(search_text), "" + + trimmed_newline_search = search_text.strip("\n") + if trimmed_newline_search and trimmed_newline_search != search_text: + attempted_descriptions.append("trimmed newline boundaries") + index = body.find(trimmed_newline_search) + if index != -1: + next_index = body.find( + trimmed_newline_search, index + len(trimmed_newline_search) + ) + if next_index != -1: + reason = ( + "SEARCH text matched multiple locations using trimmed newline " + "boundaries; increase SEARCH text length to match a longer, more specific span." + ) + return None, None, reason + return index, index + len(trimmed_newline_search), "" + + trimmed_whitespace_search = search_text.strip() + if trimmed_whitespace_search and trimmed_whitespace_search not in { + search_text, + trimmed_newline_search, + }: + attempted_descriptions.append("trimmed outer whitespace") + index = body.find(trimmed_whitespace_search) + if index != -1: + next_index = body.find( + trimmed_whitespace_search, index + len(trimmed_whitespace_search) + ) + if next_index != -1: + reason = ( + "SEARCH text matched multiple locations using trimmed outer " + "whitespace; increase SEARCH text length to match a longer, more specific span." + ) + return None, None, reason + return index, index + len(trimmed_whitespace_search), "" + + if search_text.strip(): + pattern_whitespace = _build_whitespace_pattern(search_text, allow_zero=False) + attempted_descriptions.append("normalized whitespace gaps") + matches = list(pattern_whitespace.finditer(body)) + if matches: + if len(matches) > 1: + reason = ( + "SEARCH text matched multiple locations using normalized whitespace " + "gaps; increase SEARCH text length to match a longer, more specific span." + ) + return None, None, reason + match = matches[0] + return match.start(), match.end(), "" + + pattern_relaxed = _build_whitespace_pattern(search_text, allow_zero=True) + attempted_descriptions.append("removed whitespace gaps") + matches = list(pattern_relaxed.finditer(body)) + if matches: + if len(matches) > 1: + reason = ( + "SEARCH text matched multiple locations using removed whitespace " + "gaps; increase SEARCH text length to match a longer, more specific span." + ) + return None, None, reason + match = matches[0] + return match.start(), match.end(), "" + + reason = "SEARCH text not found after attempts: " + ", ".join(attempted_descriptions) + return None, None, reason + + +def _replace_slice(body: str, start: int, end: int, replacement: str) -> str: + return body[:start] + replacement + body[end:] + + +def apply_edits( + file_contents: Dict[Path, str], + edits: List[EditInstruction], +) -> tuple[EditApplication | None, List[EditFailure]]: + updated_contents = {path: _normalize_line_endings(content) for path, content in file_contents.items()} + failures: List[EditFailure] = [] + patch_replacements: List[str] = [] + duplicate_texts: List[str] = [] + + for index, edit in enumerate(edits, start=1): + content = updated_contents[edit.file_path] + start, end, reason = _locate_search_text(content, edit.find_text) + if start is None or end is None: + failures.append( + EditFailure( + index=index, + file_path=edit.file_path, + find_text=edit.find_text, + reason=reason, + ) + ) + continue + if edit.is_duplicate: + duplicate_texts.append(edit.find_text) + continue + replacement = edit.replace_text or "" + updated_contents[edit.file_path] = _replace_slice(content, start, end, replacement) + patch_replacements.append(replacement) + + if failures: + return None, failures + + return EditApplication(updated_contents, patch_replacements, duplicate_texts), [] + + +def request_and_apply_edits( + client, + chunk_text: str, + checked_out_contents: Dict[Path, str], + checked_out_paths: Iterable[Path], + context_label: str, +) -> EditApplication: + failed_edits: List[EditFailure] | None = None + failed_formatting: str | None = None + previous_response: str | None = None + + for attempt in range(1, MAX_PATCH_ATTEMPTS + 1): + attempt_label = ( + context_label if attempt == 1 else f"{context_label} attempt {attempt}" + ) + prompt = build_edit_prompt( + chunk_text, + checked_out_contents, + failed_edits=failed_edits, + failed_formatting=failed_formatting, + previous_response=previous_response, + ) + + tool_call = request_tool_call( + client, prompt, [EDIT_TOOL_SCHEMA], f"edit {attempt_label}" + ) + previous_response = tool_call.arguments + + try: + payload = parse_tool_call_arguments(tool_call) + edit_instructions = parse_edit_instructions(payload, checked_out_paths) + except Exception as error: # noqa: BLE001 + failed_formatting = str(error) + failed_edits = None + logger.warning( + f"Edit response invalid for {attempt_label}: {error}" + ) + continue + + failed_formatting = None + application, failures = apply_edits(checked_out_contents, edit_instructions) + if not failures: + return application + failed_edits = failures + logger.info( + f"Retrying {context_label}; {len(failed_edits)} edit(s) failed to match." + ) + + raise RuntimeError( + f"Unable to apply edits for {context_label} after {MAX_PATCH_ATTEMPTS} attempt(s)." + ) diff --git a/src/spec_exploration.py b/src/spec_exploration.py new file mode 100644 index 0000000..b9e405b --- /dev/null +++ b/src/spec_exploration.py @@ -0,0 +1,282 @@ +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import Dict, Iterable, List, Tuple + +from spec_config import MAX_TOOL_ATTEMPTS, SpecConfig +from spec_llm import parse_tool_call_arguments, request_tool_call +from spec_notes import NoteRepository, ViewedNote + + +VIEW_TOOL_SCHEMA = { + "type": "function", + "name": "view_files", + "description": "Request additional files to view.", + "strict": True, + "parameters": { + "type": "object", + "properties": { + "action": {"type": "string", "enum": ["view"]}, + "files": {"type": "array", "items": {"type": "string"}}, + }, + "required": ["action", "files"], + }, +} + +CHECKOUT_TOOL_SCHEMA = { + "type": "function", + "name": "checkout_files", + "description": "Select viewed files to check out for editing.", + "strict": True, + "parameters": { + "type": "object", + "properties": { + "action": {"type": "string", "enum": ["checkout"]}, + "files": {"type": "array", "items": {"type": "string"}}, + }, + "required": ["action", "files"], + }, +} + + +@dataclass(frozen=True) +class ExplorationState: + viewed: Dict[Path, ViewedNote] + available: Dict[Path, str] + + +class ExplorationError(RuntimeError): + pass + + +def format_viewed_note(note: ViewedNote) -> str: + headings = "\n".join(f"- {heading}" for heading in note.headings) or "- " + links = ( + "\n".join(f"- [[{path.name}]] — {summary}" for path, summary in note.link_summaries) + or "- " + ) + return ( + f"## [{note.path.name}]\n\n" + f"**Summary:** {note.summary}\n\n" + f"**Headings:**\n{headings}\n\n" + f"**Links to:**\n{links}" + ) + + +def format_available_note(path: Path, summary: str) -> str: + return f"- [[{path.name}]] — {summary}" + + +def build_exploration_prompt( + chunk_text: str, + viewed_notes: Iterable[ViewedNote], + available_notes: Iterable[Tuple[Path, str]], + remaining_rounds: int, + config: SpecConfig, + feedback: str | None = None, +) -> str: + viewed_blocks = [format_viewed_note(note) for note in viewed_notes] + available_blocks = [format_available_note(path, summary) for path, summary in available_notes] + + instructions = ( + "You are exploring notes to decide which files to view next or to checkout. " + "Respond with a tool call to view_files selecting up to " + f"{config.max_files_viewed_per_round} AVAILABLE files, or call checkout_files " + "to select up to {max_checkout} VIEWED files for editing. " + "Only choose files from the provided lists." + ).format(max_checkout=config.max_files_checked_out) + + prompt = ( + "\n" + f"{instructions}\n" + "\n\n" + "\n" + f"{chunk_text}\n" + "\n\n" + "\n" + f"{'\n\n'.join(viewed_blocks) if viewed_blocks else ''}\n" + "\n\n" + "\n" + f"{'\n'.join(available_blocks) if available_blocks else ''}\n" + "\n\n" + f"{remaining_rounds}" + ) + if feedback: + prompt += f"\n\n\n{feedback}\n" + return prompt + + +def _parse_file_list(payload: dict, action: str) -> List[str]: + if payload.get("action") != action: + raise ExplorationError(f"Tool payload must include action='{action}'.") + files = payload.get("files") + if not isinstance(files, list): + raise ExplorationError("Tool payload must include a files list.") + file_names: List[str] = [] + for value in files: + if not isinstance(value, str) or not value.strip(): + raise ExplorationError("Each file entry must be a non-empty string.") + file_names.append(value.strip()) + return file_names + + +def _resolve_requested_paths( + names: Iterable[str], + mapping: Dict[str, Path], + label: str, +) -> List[Path]: + resolved: List[Path] = [] + for name in names: + key = name.lower() + path = mapping.get(key) + if path is None: + raise ExplorationError(f"Requested {label} file '{name}' is not available.") + resolved.append(path) + return resolved + + +def explore_until_checkout( + client, + chunk_text: str, + root_path: Path, + root_summary: str, + root_headings: List[str], + root_links: List[Path], + root_link_summaries: List[tuple[Path, str]], + summary_map: Dict[Path, str], + repo: NoteRepository, + config: SpecConfig, +) -> List[Path]: + viewed: Dict[Path, ViewedNote] = {} + available: Dict[Path, str] = {} + + viewed[root_path] = ViewedNote( + path=root_path, + summary=root_summary, + headings=root_headings, + links=root_links, + link_summaries=root_link_summaries, + ) + for path in root_links: + if path not in viewed and path in summary_map: + available[path] = summary_map[path] + + rounds_left = config.max_exploration_rounds + total_viewed_limit = config.max_files_viewed_total + + while rounds_left > 0: + needs_checkout = len(viewed) >= total_viewed_limit or not available + feedback = None + attempts_left = MAX_TOOL_ATTEMPTS + + while attempts_left > 0: + prompt = build_exploration_prompt( + chunk_text, + viewed.values(), + available.items(), + rounds_left, + config, + feedback=feedback, + ) + tools = [CHECKOUT_TOOL_SCHEMA] if needs_checkout else [VIEW_TOOL_SCHEMA, CHECKOUT_TOOL_SCHEMA] + tool_call = request_tool_call( + client, + prompt, + tools, + f"exploration round {config.max_exploration_rounds - rounds_left + 1}", + ) + payload = parse_tool_call_arguments(tool_call) + try: + if tool_call.name == "checkout_files": + requested = _parse_file_list(payload, "checkout") + if len(requested) > config.max_files_checked_out: + raise ExplorationError( + "Checkout request exceeds max files allowed." + ) + view_map = {path.name.lower(): path for path in viewed.keys()} + checkout_paths = _resolve_requested_paths( + requested, view_map, "viewed" + ) + if not checkout_paths: + raise ExplorationError("Checkout request must include at least one file.") + return checkout_paths + + if needs_checkout: + raise ExplorationError( + "No additional files are available to view; you must checkout." + ) + requested = _parse_file_list(payload, "view") + if len(requested) > config.max_files_viewed_per_round: + raise ExplorationError("View request exceeds max files allowed.") + available_map = {path.name.lower(): path for path in available.keys()} + requested_paths = _resolve_requested_paths( + requested, available_map, "available" + ) + except ExplorationError as error: + feedback = str(error) + attempts_left -= 1 + if attempts_left == 0: + raise + continue + + for path in requested_paths: + summary = available.pop(path) + headings = repo.get_headings(path) + links = repo.get_links(path) + link_summaries = [ + (link_path, summary_map[link_path]) + for link_path in links + if link_path in summary_map + ] + viewed[path] = ViewedNote( + path=path, + summary=summary, + headings=headings, + links=links, + link_summaries=link_summaries, + ) + for link_path in links: + if link_path not in viewed and link_path in summary_map: + available[link_path] = summary_map[link_path] + + if len(viewed) >= total_viewed_limit: + break + + rounds_left -= 1 + break + + feedback = None + attempts_left = MAX_TOOL_ATTEMPTS + while attempts_left > 0: + prompt = build_exploration_prompt( + chunk_text, + viewed.values(), + available.items(), + rounds_left, + config, + feedback=feedback, + ) + tool_call = request_tool_call( + client, + prompt, + [CHECKOUT_TOOL_SCHEMA], + "exploration checkout", + ) + payload = parse_tool_call_arguments(tool_call) + try: + requested = _parse_file_list(payload, "checkout") + if len(requested) > config.max_files_checked_out: + raise ExplorationError("Checkout request exceeds max files allowed.") + view_map = {path.name.lower(): path for path in viewed.keys()} + checkout_paths = _resolve_requested_paths(requested, view_map, "viewed") + if not checkout_paths: + raise ExplorationError("Checkout request must include at least one file.") + return checkout_paths + except ExplorationError as error: + feedback = str(error) + attempts_left -= 1 + if attempts_left == 0: + raise + + raise ExplorationError("Unable to select checkout files.") diff --git a/src/spec_llm.py b/src/spec_llm.py new file mode 100644 index 0000000..40cb049 --- /dev/null +++ b/src/spec_llm.py @@ -0,0 +1,111 @@ +from __future__ import annotations + +import json +import os +from time import sleep +from typing import Iterable + +from loguru import logger +from openai import OpenAI +from openai.types.responses import ResponseFunctionToolCall + +from spec_config import ( + DEFAULT_MAX_RETRIES, + DEFAULT_MODEL, + DEFAULT_REASONING, + ENV_API_KEY, + RETRY_BACKOFF_FACTOR, + RETRY_INITIAL_DELAY_SECONDS, +) + + +def create_openai_client() -> OpenAI: + api_key = os.getenv(ENV_API_KEY) + if not api_key: + raise RuntimeError( + f"Environment variable {ENV_API_KEY} is required for GPT access." + ) + return OpenAI(api_key=api_key) + + +def execute_with_retry( + operation, + description: str, + max_attempts: int = DEFAULT_MAX_RETRIES, + initial_delay_seconds: float = RETRY_INITIAL_DELAY_SECONDS, + backoff_factor: float = RETRY_BACKOFF_FACTOR, +): + attempt = 1 + delay = initial_delay_seconds + while True: + try: + return operation() + except Exception as error: + if attempt >= max_attempts: + logger.exception( + f"OpenAI {description} failed after {max_attempts} attempt(s): {error}" + ) + raise + logger.warning( + f"OpenAI {description} attempt {attempt} failed: {error}. Retrying in {delay:.1f}s." + ) + sleep(delay) + attempt += 1 + delay *= backoff_factor + + +def request_text(client: OpenAI, prompt: str, context_label: str) -> str: + def perform_request() -> str: + response = client.responses.create( + model=DEFAULT_MODEL, + reasoning=DEFAULT_REASONING, + input=prompt, + ) + if response.error: + raise RuntimeError(f"OpenAI error for {context_label}: {response.error}") + output_text = response.output_text + if not output_text.strip(): + raise RuntimeError(f"Received empty response for {context_label}.") + return output_text.strip() + + return execute_with_retry(perform_request, context_label) + + +def request_tool_call( + client: OpenAI, prompt: str, tools: Iterable[dict], context_label: str +) -> ResponseFunctionToolCall: + def perform_request() -> ResponseFunctionToolCall: + response = client.responses.create( + model=DEFAULT_MODEL, + reasoning=DEFAULT_REASONING, + input=prompt, + tools=list(tools), + tool_choice="required", + parallel_tool_calls=False, + ) + if response.error: + raise RuntimeError(f"OpenAI error for {context_label}: {response.error}") + tool_calls = [item for item in response.output if item.type == "function_call"] + if not tool_calls: + raise RuntimeError(f"No tool call returned for {context_label}.") + if len(tool_calls) > 1: + raise RuntimeError( + f"Expected a single tool call for {context_label}, got {len(tool_calls)}." + ) + return tool_calls[0] + + return execute_with_retry(perform_request, context_label) + + +def parse_tool_call_arguments(call: ResponseFunctionToolCall) -> dict: + if not call.arguments: + raise RuntimeError(f"Tool call {call.name} missing arguments.") + try: + payload = json.loads(call.arguments) + except json.JSONDecodeError as error: + raise RuntimeError( + f"Tool call {call.name} arguments are not valid JSON: {error}" + ) from error + if not isinstance(payload, dict): + raise RuntimeError(f"Tool call {call.name} arguments must be a JSON object.") + return payload diff --git a/src/spec_logging.py b/src/spec_logging.py new file mode 100644 index 0000000..aeba1bb --- /dev/null +++ b/src/spec_logging.py @@ -0,0 +1,24 @@ +from __future__ import annotations + +import sys +from pathlib import Path + +from loguru import logger + +from spec_config import LOG_FILE_ROTATION_BYTES + + +def configure_logging(log_path: Path) -> None: + logger.remove() + logger.add(sys.stderr, level="INFO", enqueue=False) + try: + log_path.parent.mkdir(parents=True, exist_ok=True) + except OSError as error: + raise RuntimeError(f"Failed to prepare log directory {log_path.parent}: {error}") from error + logger.add( + log_path, + level="DEBUG", + rotation=LOG_FILE_ROTATION_BYTES, + enqueue=False, + encoding="utf-8", + ) diff --git a/src/spec_markdown.py b/src/spec_markdown.py new file mode 100644 index 0000000..5584488 --- /dev/null +++ b/src/spec_markdown.py @@ -0,0 +1,74 @@ +from __future__ import annotations + +import re +from typing import List, Tuple + + +WIKILINK_PATTERN = re.compile(r"\[\[([^\]|#]+)(?:#[^\]|]+)?(?:\|[^\]]+)?\]\]") +HEADING_PATTERN = re.compile(r"^(#{1,6})\s+(.+?)\s*$") + + +def split_document_sections(content: str, scratchpad_heading: str) -> Tuple[str, str]: + if scratchpad_heading not in content: + raise ValueError(f"Document must contain the heading '{scratchpad_heading}'.") + heading_index = content.index(scratchpad_heading) + body = content[:heading_index].rstrip() + scratchpad = content[heading_index + len(scratchpad_heading) :].lstrip("\n") + return body, scratchpad + + +def normalize_paragraphs(text: str) -> List[str]: + stripped_text = text.strip() + if not stripped_text: + return [] + return [ + block.strip() for block in re.split(r"\n\s*\n", stripped_text) if block.strip() + ] + + +def count_words(text: str) -> int: + return len(text.split()) + + +def extract_headings(content: str) -> List[str]: + headings: List[str] = [] + for line in content.splitlines(): + match = HEADING_PATTERN.match(line) + if match: + hashes, title = match.groups() + headings.append(f"{hashes} {title.strip()}") + return headings + + +def extract_wikilinks(content: str) -> List[str]: + targets: List[str] = [] + for match in WIKILINK_PATTERN.finditer(content): + target = match.group(1).strip() + if target: + targets.append(target) + return targets + + +def build_document(body: str, scratchpad_heading: str, scratchpad_paragraphs: List[str]) -> str: + trimmed_body = body.rstrip() + parts = [trimmed_body, scratchpad_heading] + if scratchpad_paragraphs: + scratchpad_text = "\n\n".join(scratchpad_paragraphs).rstrip() + parts.append(scratchpad_text) + document = "\n\n".join(part for part in parts if part) + if not document.endswith("\n"): + document += "\n" + return document + + +def format_duration(seconds: float) -> str: + remaining_seconds = max(0, int(round(seconds))) + hours, remainder = divmod(remaining_seconds, 3600) + minutes, seconds = divmod(remainder, 60) + parts: List[str] = [] + if hours: + parts.append(f"{hours}h") + if hours or minutes: + parts.append(f"{minutes}m") + parts.append(f"{seconds}s") + return " ".join(parts) diff --git a/src/spec_notes.py b/src/spec_notes.py new file mode 100644 index 0000000..6ceb34d --- /dev/null +++ b/src/spec_notes.py @@ -0,0 +1,87 @@ +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import Dict, Iterable, List, Optional, Set + +from spec_markdown import count_words, extract_headings, extract_wikilinks + + +@dataclass(frozen=True) +class ViewedNote: + path: Path + summary: str + headings: List[str] + links: List[Path] + link_summaries: List[tuple[Path, str]] + + +class NoteRepository: + def __init__(self, root_path: Path, root_body: str, notes_dir: Path) -> None: + self._root_path = root_path + self._root_body = root_body + self._notes_dir = notes_dir + self._content_cache: Dict[Path, str] = {} + self._file_index = self._build_file_index() + + def _build_file_index(self) -> Dict[str, Path]: + mapping: Dict[str, Path] = {} + for path in self._notes_dir.iterdir(): + if path.is_file() and path.suffix.lower() == ".md": + mapping[path.name.lower()] = path + return mapping + + def resolve_link(self, link_text: str) -> Optional[Path]: + target = link_text.strip() + if not target: + return None + if not target.lower().endswith(".md"): + target = f"{target}.md" + return self._file_index.get(target.lower()) + + def get_note_content(self, path: Path) -> str: + if path == self._root_path: + return self._root_body + cached = self._content_cache.get(path) + if cached is not None: + return cached + content = path.read_text(encoding="utf-8") + self._content_cache[path] = content + return content + + def get_headings(self, path: Path) -> List[str]: + return extract_headings(self.get_note_content(path)) + + def get_links(self, path: Path) -> List[Path]: + links: List[Path] = [] + for target in extract_wikilinks(self.get_note_content(path)): + resolved = self.resolve_link(target) + if resolved is not None: + links.append(resolved) + return links + + def get_word_count(self, path: Path) -> int: + return count_words(self.get_note_content(path)) + + def is_index_note(self, path: Path, index_suffix: str) -> bool: + return path.name.lower().endswith(index_suffix.lower()) + + def iter_reachable_paths(self) -> List[Path]: + visited: Set[Path] = set() + stack: List[Path] = [self._root_path] + while stack: + path = stack.pop() + if path in visited: + continue + visited.add(path) + for link in self.get_links(path): + if link not in visited: + stack.append(link) + return list(visited) + + def set_root_body(self, body: str) -> None: + self._root_body = body + self._content_cache.pop(self._root_path, None) + + def invalidate_content(self, path: Path) -> None: + self._content_cache.pop(path, None) diff --git a/src/spec_summary.py b/src/spec_summary.py new file mode 100644 index 0000000..9370640 --- /dev/null +++ b/src/spec_summary.py @@ -0,0 +1,204 @@ +from __future__ import annotations + +import hashlib +import json +import os +from concurrent.futures import Future, ThreadPoolExecutor +from dataclasses import dataclass +from pathlib import Path +from threading import Lock +from typing import Dict, Iterable, List + +from spec_config import SpecConfig, default_summary_cache_path +from spec_llm import request_text +from spec_markdown import extract_wikilinks +from spec_notes import NoteRepository + + +@dataclass(frozen=True) +class SummaryRecord: + content_hash: str + summary: str + + +class SummaryCache: + def __init__(self, cache_path: Path) -> None: + self._path = cache_path + self._lock = Lock() + self._data: Dict[str, SummaryRecord] = {} + self._load() + + def _load(self) -> None: + if not self._path.exists(): + return + raw = self._path.read_text(encoding="utf-8") + if not raw.strip(): + return + data = json.loads(raw) + if not isinstance(data, dict): + raise RuntimeError("Summary cache must contain a JSON object.") + for key, value in data.items(): + if not isinstance(value, dict): + continue + content_hash = value.get("content_hash") + summary = value.get("summary") + if isinstance(content_hash, str) and isinstance(summary, str): + self._data[key] = SummaryRecord(content_hash, summary) + + def get(self, path: Path, content_hash: str) -> str | None: + record = self._data.get(str(path)) + if record and record.content_hash == content_hash: + return record.summary + return None + + def set(self, path: Path, content_hash: str, summary: str) -> None: + with self._lock: + self._data[str(path)] = SummaryRecord(content_hash, summary) + self._path.parent.mkdir(parents=True, exist_ok=True) + payload = { + key: {"content_hash": record.content_hash, "summary": record.summary} + for key, record in self._data.items() + } + self._path.write_text( + json.dumps(payload, ensure_ascii=True, indent=2), encoding="utf-8" + ) + + def invalidate(self, path: Path) -> None: + with self._lock: + if str(path) in self._data: + self._data.pop(str(path)) + self._path.parent.mkdir(parents=True, exist_ok=True) + payload = { + key: {"content_hash": record.content_hash, "summary": record.summary} + for key, record in self._data.items() + } + self._path.write_text( + json.dumps(payload, ensure_ascii=True, indent=2), encoding="utf-8" + ) + + +def _hash_content(content: str) -> str: + return hashlib.sha256(content.encode("utf-8")).hexdigest() + + +def _default_summary_workers() -> int: + cpu_count = os.cpu_count() or 4 + return max(4, min(32, cpu_count * 4)) + + +class SummaryService: + def __init__( + self, + repo: NoteRepository, + client, + config: SpecConfig, + cache_path: Path | None = None, + ) -> None: + self._repo = repo + self._client = client + self._config = config + self._cache = SummaryCache(cache_path or default_summary_cache_path()) + self._executor = ThreadPoolExecutor(max_workers=_default_summary_workers()) + self._lock = Lock() + self._inflight: Dict[Path, Future[str]] = {} + + def shutdown(self) -> None: + self._executor.shutdown(wait=True) + + def invalidate(self, path: Path) -> None: + with self._lock: + self._inflight.pop(path, None) + self._cache.invalidate(path) + + def get_summaries(self, paths: Iterable[Path]) -> Dict[Path, str]: + futures = {path: self._ensure_future(path) for path in paths} + return {path: future.result() for path, future in futures.items()} + + def get_summary(self, path: Path) -> str: + return self._ensure_future(path).result() + + def _ensure_future(self, path: Path) -> Future[str]: + with self._lock: + existing = self._inflight.get(path) + if existing is not None: + return existing + future: Future[str] = self._executor.submit(self._compute_summary, path) + self._inflight[path] = future + return future + + def _compute_summary(self, path: Path) -> str: + try: + return self._compute_summary_inner(path, stack=[], allow_inflight_wait=False) + finally: + with self._lock: + self._inflight.pop(path, None) + + def _compute_summary_inner( + self, path: Path, stack: List[Path], allow_inflight_wait: bool = True + ) -> str: + if path in stack: + cycle = " -> ".join(item.name for item in stack + [path]) + raise RuntimeError(f"Cycle detected while summarizing index notes: {cycle}") + + if allow_inflight_wait: + with self._lock: + inflight = self._inflight.get(path) + if inflight is not None: + return inflight.result() + + content = self._repo.get_note_content(path) + content_hash = _hash_content(content) + cached = self._cache.get(path, content_hash) + if cached is not None: + return cached + + stack.append(path) + try: + if self._repo.is_index_note(path, self._config.index_filename_suffix): + summary = self._summarize_index_note(path, content, stack) + else: + summary = self._summarize_standard_note(path, content) + finally: + stack.pop() + + self._cache.set(path, content_hash, summary) + return summary + + def _summarize_standard_note(self, path: Path, content: str) -> str: + prompt = ( + "Generate a {min_words}-{max_words} word summary of this note's content.\n" + "Focus on: main topics, key claims, what questions it answers.\n\n" + "\n{content}\n" + ).format( + min_words=self._config.summary_target_words_min, + max_words=self._config.summary_target_words_max, + content=content, + ) + return request_text(self._client, prompt, f"summary {path.name}") + + def _summarize_index_note(self, path: Path, content: str, stack: List[Path]) -> str: + linked_paths = [] + for target in extract_wikilinks(content): + resolved = self._repo.resolve_link(target) + if resolved is not None: + linked_paths.append(resolved) + + summaries: List[str] = [] + for linked_path in linked_paths: + summaries.append( + self._compute_summary_inner( + linked_path, stack, allow_inflight_wait=False + ) + ) + + joined_summaries = "\n\n".join(summaries) if summaries else "" + prompt = ( + "Generate a summary based on these summaries of linked notes:\n" + "{summaries}\n\n" + "Synthesize into {min_words}-{max_words} words describing what this index covers." + ).format( + summaries=joined_summaries, + min_words=self._config.summary_target_words_min, + max_words=self._config.summary_target_words_max, + ) + return request_text(self._client, prompt, f"summary {path.name}") diff --git a/src/spec_verification.py b/src/spec_verification.py new file mode 100644 index 0000000..5dc267e --- /dev/null +++ b/src/spec_verification.py @@ -0,0 +1,327 @@ +from __future__ import annotations + +import json +import shutil +import subprocess +from concurrent.futures import ThreadPoolExecutor +from dataclasses import dataclass +from pathlib import Path +from threading import Event, Lock, Thread +from typing import Any, List, Sequence +from uuid import uuid4 + +from loguru import logger + +from spec_config import MAX_CONCURRENT_VERIFICATIONS, default_pending_prompts_path +from spec_llm import request_text + + +NOTIFY_SEND_PATH = shutil.which("notify-send") +_NOTIFY_SEND_UNAVAILABLE_WARNING_EMITTED = False + + +def notify_missing_verification( + chunk_index: int, total_chunks: int, assessment: str +) -> None: + global _NOTIFY_SEND_UNAVAILABLE_WARNING_EMITTED + title = "Integration verification missing content" + body = f"Chunk {chunk_index + 1}/{total_chunks}: {assessment}" + if NOTIFY_SEND_PATH: + try: + subprocess.run( + [ + NOTIFY_SEND_PATH, + "--app-name=IntegrateNotes", + title, + body, + ], + check=True, + ) + except Exception as error: + logger.warning( + f"notify-send failed for verification chunk {chunk_index + 1}: {error}" + ) + else: + if not _NOTIFY_SEND_UNAVAILABLE_WARNING_EMITTED: + logger.warning( + "notify-send not available; desktop alerts for verification issues disabled." + ) + _NOTIFY_SEND_UNAVAILABLE_WARNING_EMITTED = True + + +@dataclass(frozen=True) +class DuplicateEvidence: + body_text: str + + +class VerificationManager: + def __init__(self, client, target_file: Path) -> None: + self.client = client + self.pending_path = default_pending_prompts_path() + self.lock = Lock() + self.active_lock = Lock() + self.active_ids: set[str] = set() + self.executor = ThreadPoolExecutor(max_workers=MAX_CONCURRENT_VERIFICATIONS) + self.new_prompt_event = Event() + self.stop_requested = False + self.tracked_file_name = Path(target_file).resolve().name + self.worker = Thread( + target=self._run, + name="VerificationManager", + daemon=True, + ) + self.worker.start() + + def enqueue_prompt( + self, + prompt: str, + context_label: str | None, + chunk_index: int | None, + total_chunks: int | None, + ) -> None: + if not isinstance(prompt, str) or not prompt.strip(): + raise ValueError("Verification prompt must be a non-empty string.") + + entry = { + "id": str(uuid4()), + "prompt": prompt, + "context_label": context_label, + "chunk_index": chunk_index, + "total_chunks": total_chunks, + "file_name": self.tracked_file_name, + } + with self.lock: + entries = self._read_entries_locked() + entries.append(entry) + self._write_entries_locked(entries) + self.new_prompt_event.set() + + def shutdown(self) -> None: + self.stop_requested = True + self.new_prompt_event.set() + if self.worker.is_alive(): + self.worker.join() + self.executor.shutdown(wait=True) + + def _run(self) -> None: + while True: + try: + self._dispatch_pending() + except Exception as error: + logger.exception( + f"Verification dispatcher encountered an error: {error}" + ) + if self.stop_requested and not self._has_pending_work(): + break + self.new_prompt_event.wait(timeout=0.5) + self.new_prompt_event.clear() + + def _dispatch_pending(self) -> None: + with self.lock: + all_entries = self._read_entries_locked() + entries = self._entries_for_current_file_locked(all_entries) + + for entry in entries: + entry_id = entry.get("id") + if not entry_id: + continue + with self.active_lock: + if entry_id in self.active_ids: + continue + self.active_ids.add(entry_id) + + future = self.executor.submit(self._send_prompt, entry) + future.add_done_callback( + lambda fut, data=entry: self._handle_result(data, fut) + ) + + def _send_prompt(self, entry: dict[str, Any]) -> str: + context_label = entry.get("context_label") or "verification" + prompt = entry["prompt"] + return request_text(self.client, prompt, f"verification {context_label}") + + def _handle_result(self, entry: dict[str, Any], future) -> None: + entry_id = entry.get("id") + try: + assessment = future.result() + except Exception as error: # noqa: BLE001 + context_label = entry.get("context_label") or "verification" + logger.exception(f"Verification for {context_label} failed: {error}") + if entry_id: + with self.active_lock: + self.active_ids.discard(entry_id) + self.new_prompt_event.set() + return + + self._log_assessment(entry, assessment) + + if entry_id: + self._remove_entry(entry_id) + with self.active_lock: + self.active_ids.discard(entry_id) + + self.new_prompt_event.set() + + def _log_assessment(self, entry: dict[str, Any], assessment: str) -> None: + chunk_index = entry.get("chunk_index") + total_chunks = entry.get("total_chunks") + context_label = entry.get("context_label") or "verification" + file_name = entry.get("file_name") + + if not file_name: + raise RuntimeError( + "Verification entry missing required file_name; pending prompts file may be corrupted." + ) + + base_header = f'Verification "{file_name}"' + + if ( + isinstance(chunk_index, int) + and isinstance(total_chunks, int) + and 0 <= chunk_index < total_chunks + ): + if "MISSING" in assessment: + notify_missing_verification(chunk_index, total_chunks, assessment) + chunk_header = f"{base_header}:" + if assessment.startswith(chunk_header): + logger.info(assessment) + else: + logger.info(f"{chunk_header}\n{assessment}") + else: + if context_label != "verification": + header = f"{base_header} ({context_label}):" + else: + header = f"{base_header}:" + if assessment.startswith(header): + logger.info(assessment) + else: + logger.info(f"{header}\n{assessment}") + + def _remove_entry(self, entry_id: str) -> None: + with self.lock: + entries = self._read_entries_locked() + remaining = [item for item in entries if item.get("id") != entry_id] + self._write_entries_locked(remaining) + + def _read_entries_locked(self) -> List[dict[str, Any]]: + if not self.pending_path.exists(): + return [] + raw = self.pending_path.read_text(encoding="utf-8") + if not raw.strip(): + return [] + try: + data = json.loads(raw) + except json.JSONDecodeError as error: + raise RuntimeError( + f"Pending verification prompts file {self.pending_path} is corrupted: {error}" + ) from error + if not isinstance(data, list): + raise RuntimeError( + f"Pending verification prompts file {self.pending_path} must contain a list." + ) + return data + + def _write_entries_locked(self, entries: List[dict[str, Any]]) -> None: + self.pending_path.parent.mkdir(parents=True, exist_ok=True) + payload = json.dumps(entries, ensure_ascii=True, indent=2) + self.pending_path.write_text(payload, encoding="utf-8") + + def _has_pending_work(self) -> bool: + with self.lock: + entries = self._read_entries_locked() + has_entries = bool(self._entries_for_current_file_locked(entries)) + with self.active_lock: + has_active = bool(self.active_ids) + return has_entries or has_active + + def _entries_for_current_file_locked( + self, entries: List[dict[str, Any]] + ) -> List[dict[str, Any]]: + invalid_entries: List[dict[str, Any]] = [] + relevant_entries: List[dict[str, Any]] = [] + + for entry in entries: + file_name = entry.get("file_name") + entry_id = entry.get("id") + if not file_name or not entry_id: + invalid_entries.append(entry) + continue + if file_name == self.tracked_file_name: + relevant_entries.append(entry) + + if invalid_entries: + invalid_count = len(invalid_entries) + suffix = "y" if invalid_count == 1 else "ies" + logger.warning( + f"Removed {invalid_count} invalid verification prompt entr{suffix} missing file metadata or IDs." + ) + cleaned_entries = [ + entry for entry in entries if entry not in invalid_entries + ] + self._write_entries_locked(cleaned_entries) + + return relevant_entries + + +def build_verification_prompt( + chunk_text: str, + patch_replacements: Sequence[str], + duplicate_texts: Sequence[str], +) -> str: + response_instructions = ( + "Report whether any note content is missing or materially altered." + " Respond with a concise single paragraph beginning with 'OK -' if everything is covered" + " or 'MISSING -' followed by details of any omissions." + " Separate each omission by two newlines and for each omission, provide the following:\n" + ' Notes:"..."\n' + ' Body:"..."\n' + ' Explanation: "..."\n' + ' Proposed Fix: "..."\n' + "Quote the exact text from the notes chunk containing the missing detail and quote the exact passage from the patch replacements or duplicate evidence that should cover it (or state Body:\"\" if nothing is relevant)." + " Explain precisely what information is still missing or altered without omitting any nuance." + ) + + if patch_replacements: + replacement_sections = [] + for index, replacement_text in enumerate(patch_replacements, start=1): + replacement_sections.append( + f"[Patch {index} Replacement]\n{replacement_text}" + ) + replacements_block = "\n\n".join(replacement_sections) + else: + replacements_block = "" + + if duplicate_texts: + duplication_sections = [] + for index, body_text in enumerate(duplicate_texts, start=1): + duplication_sections.append( + f"[Duplicate {index} Evidence]\nBody:\n{body_text}" + ) + duplications_block = "\n\n".join(duplication_sections) + else: + duplications_block = "" + + sections = [ + ( + "" + "You are verifying that every idea/point/concept/argument/detail/url/[[wikilink]]/diagram etc. " + "from the provided notes chunk has been integrated into the document body." + " Use the patch replacements to understand what will be inserted or rewritten." + " Duplicate evidence is existing body text claimed to already cover notes." + " If duplicate evidence does not fully cover the notes text, treat the missing detail as missing." + "" + ), + f"\n{chunk_text}\n", + f"\n{replacements_block}\n", + f"\n{duplications_block}\n", + f"\n{response_instructions}\n", + ] + return "\n\n\n\n\n".join(sections) + + +def format_verification_assessment(assessment: str) -> str: + return ( + assessment.replace(" - Notes:", "\nNotes:") + .replace(" Body:", "\nBody:") + .replace(" Explanation:", "\nExplanation:") + ) From 109de7cfc87dc03127106d304825001f77497097 Mon Sep 17 00:00:00 2001 From: distbit Date: Tue, 6 Jan 2026 09:15:00 +0700 Subject: [PATCH 05/27] chore: clear interview --- INTERVIEW.md | 101 --------------------------------------------------- 1 file changed, 101 deletions(-) diff --git a/INTERVIEW.md b/INTERVIEW.md index f4fdc7c..e69de29 100644 --- a/INTERVIEW.md +++ b/INTERVIEW.md @@ -1,101 +0,0 @@ -# SPEC.md Implementation - -## Scope & entry point -> Should the SPEC flow fully replace the current scratchpad-based integration in `src/integrate_notes.py`, or should it live as a new CLI/module (e.g., `src/zet_integrator.py`) with its own entry point? - -the inbox referred to in the spec i.e. where the text is coming from should still be the scratchpad of the file specified in the cli, the same way it is currently. the "root" note mentioned in the spec is just whichever file is passed to integrate_notes.py i.e. the file which we read the scratchpad of. - -> If it replaces the current flow, should we remove the existing scratchpad/grouping CLI flags and related logic entirely (per “no backward compatibility”), or keep any pieces (which ones)? - -ok actually so what i want to do is keep the old version of integrate_notes.py untouched, but implement the SPEC.md in a new py file, and in this file, remove all grouping instructions related logic. but keep scratchpad stuff, as that is where the notes which are being chunked/integrated are coming from. - -## Notes repository & config -> What are the concrete paths for `notes_directory`, `root_file`, and `inbox_file` in your environment (relative to repo or absolute), and should they be set via `config.json`, CLI flags, or environment variables (and which should take precedence)? - -inbox file is just the scratchpad of root file, and root file is just the file provided via the cli, and notes directory is just whatever directory root file is in. inbox file is not a separate file, despite what the spec says. - -> For `index_filename_pattern`, is the simple substring match in the spec (“index”) correct for your notes, or do you want a stricter rule (e.g., suffix/prefix/regex)? - -if it ends in index.md, it is an index file. - -> Should we treat non-note markdown files in the notes directory (templates, archives, etc.) as excluded by default? If yes, how do you want them identified (folder name, filename pattern, front matter flag)? - -files are only "included" if they linked to, directly or indirectly (through another file) from the root file. only markdown files can be linked to. the fact that you ask me this seems to indicate you have some confusion though, as i don't see a need to exclude files just because they are in the dir. we don't scan all files in the dir. we only look at linked files and derive their path from the link text, .md and the root file dir (noting that links are case insensitive though) - -## Chunking -> When numbering paragraphs in the inbox, do you want paragraphs split strictly on blank lines (current `normalize_paragraphs` behavior), or should we treat other separators (e.g., horizontal rules) as paragraph boundaries too? - -just blank lines - -> For non-contiguous chunk groups returned by the model, should we preserve original paragraph order when assembling each chunk, or should we follow the order returned by the model? - -the latter. - -> For the “15 randomly sampled filenames (>300 words)” calibration step, should the sample be deterministic (seeded) for reproducibility, and should word counts ignore front matter/code blocks? - -doesn't need to be seeded. doesn't need to ignore front matter. note that these must only be files which are linked to (recursively/indirectly or directly) from the root note. not just any files in the dir. - -## Summaries & cache -> For summary cache storage, should we use the spec’s `.summary_cache.json` at repo root, or do you prefer a different location/format (and should it be tracked in git or ignored)? - -no put it outside the directory to avoid creating a mess. - -> For index-note summaries (summaries of linked notes), should we include links that resolve outside `notes_directory`, or only within it? How should broken/missing links be handled (error vs. skip)? - -skip links for which no notes file exists. links are NOT capable of resolving outside of the notes directory, as the link text just specifies the file name and the directory is always implicitly notes directory. - -> Should summary generation happen as a separate command (preprocessing), or on-demand during integration if a needed summary is stale/missing? - -on-demand. but make sure it is maximally parallelised, to avoid needing to wait a long time. - -## Markdown parsing & links -> Which link syntaxes should count as “outgoing links” for exploration: `[[wikilink]]`, `[text](file.md)`, bare `file.md`, or something else? Any special handling for anchors like `[[note#section]]`? - -only wikilinks. if it has a #section in the wikilink, just ignore the section. too complex. - -> For headings extraction, should we only parse ATX `#` headings (ignore Setext), and should headings inside code blocks be ignored? - -ignore setext. do not ignore headings inside code blocks. too complex - -## LLM interaction & tools -> Do you want true tool calling via OpenAI Responses API tools (structured `action: view/checkout/edit`), or is strict JSON schema parsing of a normal response sufficient as long as the schema matches the spec? - -yes use tools. - -> Which model and reasoning level should we use for chunking, summaries, exploration, and edit phases? Should they all be the same (currently `gpt-5.2`/`medium`) or different per phase? - -yes use medium for all. - -> When an edit validation fails (0 or >1 matches), should we send the model just the failure details, or also include the specific matching candidates/snippets from the file to help it correct? - -just failure details. - -## Editing & validation -> The spec’s whitespace-normalized matching treats runs of whitespace as equivalent. Should we still allow the exact/trimmed match fallbacks currently used in `_locate_search_text`, or switch strictly to the whitespace-normalized approach only? - -both (but simplify if possible). be quite permissive here. we like to avoid unnecessary matching errors. - -> For `is_duplicate: true` edits, do you want them surfaced in output/logs (and how), or just kept internally for audit? - -preserve the existing verification logic, but update it to support these new edit request formats. - -## Logging & runtime behavior -> Where should logs live (the spec doesn’t say, but project rules say “move logs outside src/”)? Is `./logs/integrate_notes.log` acceptable, or do you want a different path/rotation policy? - -yes that is ok. - -> Should the integration run be single-threaded for determinism, or is limited concurrency acceptable for summary generation/exploration? - -concurrency is desired for any api calls. - - -also pls split up the code, including the existing logic, across multiple semantically-meaningful files each focusing on a different area, as the current file is wayy too long. - -## Follow-ups -> You asked to keep the old `integrate_notes.py` untouched but also to split the existing logic into multiple files. Which should take priority: (a) leave `integrate_notes.py` as-is and only add new modules for the SPEC flow, or (b) refactor the existing logic into multiple files while keeping behavior unchanged? - -keep the old one. but duplicate it and refactor + implement new logic in this duplicate version. - -> For the summary cache “outside the directory,” do you want it outside the notes directory but still inside the repo (e.g., `./.summary_cache.json`), or in a user-level cache dir (e.g., `~/.cache/integrate_notes/summary_cache.json`)? If user-level, which exact path should we use on Fedora? - -user level. up to you re: path From e5de72bb425a419109a0794e1298690e75121537 Mon Sep 17 00:00:00 2001 From: distbit Date: Tue, 6 Jan 2026 09:40:18 +0700 Subject: [PATCH 06/27] Harden exploration parsing and summary concurrency --- src/spec_exploration.py | 38 +++++++++++--- src/spec_summary.py | 107 ++++++++++++++++++++++++++-------------- 2 files changed, 102 insertions(+), 43 deletions(-) diff --git a/src/spec_exploration.py b/src/spec_exploration.py index b9e405b..c8c1c2f 100644 --- a/src/spec_exploration.py +++ b/src/spec_exploration.py @@ -51,10 +51,9 @@ class ExplorationError(RuntimeError): def format_viewed_note(note: ViewedNote) -> str: - headings = "\n".join(f"- {heading}" for heading in note.headings) or "- " - links = ( - "\n".join(f"- [[{path.name}]] — {summary}" for path, summary in note.link_summaries) - or "- " + headings = "\n".join(f"- {heading}" for heading in note.headings) + links = "\n".join( + f"- [[{path.name}]] — {summary}" for path, summary in note.link_summaries ) return ( f"## [{note.path.name}]\n\n" @@ -107,6 +106,33 @@ def build_exploration_prompt( return prompt +def _normalize_file_name(value: str) -> str: + trimmed = value.strip() + if trimmed.startswith("[[") and trimmed.endswith("]]"): + trimmed = trimmed[2:-2].strip() + if "|" in trimmed: + trimmed = trimmed.split("|", 1)[0].strip() + if "#" in trimmed: + trimmed = trimmed.split("#", 1)[0].strip() + if not trimmed: + raise ExplorationError("File reference cannot be empty.") + if not trimmed.lower().endswith(".md"): + trimmed = f"{trimmed}.md" + return trimmed + + +def _dedupe_preserve_order(values: Iterable[str]) -> List[str]: + seen: set[str] = set() + result: List[str] = [] + for value in values: + key = value.lower() + if key in seen: + continue + seen.add(key) + result.append(value) + return result + + def _parse_file_list(payload: dict, action: str) -> List[str]: if payload.get("action") != action: raise ExplorationError(f"Tool payload must include action='{action}'.") @@ -117,8 +143,8 @@ def _parse_file_list(payload: dict, action: str) -> List[str]: for value in files: if not isinstance(value, str) or not value.strip(): raise ExplorationError("Each file entry must be a non-empty string.") - file_names.append(value.strip()) - return file_names + file_names.append(_normalize_file_name(value)) + return _dedupe_preserve_order(file_names) def _resolve_requested_paths( diff --git a/src/spec_summary.py b/src/spec_summary.py index 9370640..eca8597 100644 --- a/src/spec_summary.py +++ b/src/spec_summary.py @@ -111,56 +111,59 @@ def invalidate(self, path: Path) -> None: self._cache.invalidate(path) def get_summaries(self, paths: Iterable[Path]) -> Dict[Path, str]: - futures = {path: self._ensure_future(path) for path in paths} - return {path: future.result() for path, future in futures.items()} + unique_paths = list(dict.fromkeys(paths)) + standard_paths: List[Path] = [] + index_paths: List[Path] = [] + for path in unique_paths: + if self._repo.is_index_note(path, self._config.index_filename_suffix): + index_paths.append(path) + else: + standard_paths.append(path) + + futures = {path: self._ensure_future(path) for path in standard_paths} + summaries: Dict[Path, str] = { + path: future.result() for path, future in futures.items() + } + + for path in index_paths: + summaries[path] = self._compute_index_summary(path, stack=[]) + + return summaries def get_summary(self, path: Path) -> str: + if self._repo.is_index_note(path, self._config.index_filename_suffix): + return self._compute_index_summary(path, stack=[]) return self._ensure_future(path).result() def _ensure_future(self, path: Path) -> Future[str]: + if self._repo.is_index_note(path, self._config.index_filename_suffix): + raise RuntimeError( + f"Index note summaries must be computed synchronously: {path.name}." + ) with self._lock: existing = self._inflight.get(path) if existing is not None: return existing - future: Future[str] = self._executor.submit(self._compute_summary, path) + future: Future[str] = self._executor.submit( + self._compute_standard_summary, path + ) self._inflight[path] = future return future - def _compute_summary(self, path: Path) -> str: + def _compute_standard_summary(self, path: Path) -> str: try: - return self._compute_summary_inner(path, stack=[], allow_inflight_wait=False) + return self._compute_standard_summary_inner(path) finally: with self._lock: self._inflight.pop(path, None) - def _compute_summary_inner( - self, path: Path, stack: List[Path], allow_inflight_wait: bool = True - ) -> str: - if path in stack: - cycle = " -> ".join(item.name for item in stack + [path]) - raise RuntimeError(f"Cycle detected while summarizing index notes: {cycle}") - - if allow_inflight_wait: - with self._lock: - inflight = self._inflight.get(path) - if inflight is not None: - return inflight.result() - + def _compute_standard_summary_inner(self, path: Path) -> str: content = self._repo.get_note_content(path) content_hash = _hash_content(content) cached = self._cache.get(path, content_hash) if cached is not None: return cached - - stack.append(path) - try: - if self._repo.is_index_note(path, self._config.index_filename_suffix): - summary = self._summarize_index_note(path, content, stack) - else: - summary = self._summarize_standard_note(path, content) - finally: - stack.pop() - + summary = self._summarize_standard_note(path, content) self._cache.set(path, content_hash, summary) return summary @@ -176,22 +179,52 @@ def _summarize_standard_note(self, path: Path, content: str) -> str: ) return request_text(self._client, prompt, f"summary {path.name}") + def _compute_index_summary(self, path: Path, stack: List[Path]) -> str: + if path in stack: + cycle = " -> ".join(item.name for item in stack + [path]) + raise RuntimeError(f"Cycle detected while summarizing index notes: {cycle}") + + content = self._repo.get_note_content(path) + content_hash = _hash_content(content) + cached = self._cache.get(path, content_hash) + if cached is not None: + return cached + + stack.append(path) + try: + summary = self._summarize_index_note(path, content, stack) + finally: + stack.pop() + + self._cache.set(path, content_hash, summary) + return summary + def _summarize_index_note(self, path: Path, content: str, stack: List[Path]) -> str: - linked_paths = [] + linked_paths: List[Path] = [] + seen: set[Path] = set() for target in extract_wikilinks(content): resolved = self._repo.resolve_link(target) - if resolved is not None: + if resolved is not None and resolved not in seen: + seen.add(resolved) linked_paths.append(resolved) - summaries: List[str] = [] + standard_paths: List[Path] = [] + index_paths: List[Path] = [] for linked_path in linked_paths: - summaries.append( - self._compute_summary_inner( - linked_path, stack, allow_inflight_wait=False - ) - ) + if self._repo.is_index_note(linked_path, self._config.index_filename_suffix): + index_paths.append(linked_path) + else: + standard_paths.append(linked_path) + + futures = {linked_path: self._ensure_future(linked_path) for linked_path in standard_paths} + + summaries: List[str] = [] + for linked_path in index_paths: + summaries.append(self._compute_index_summary(linked_path, stack)) + for linked_path, future in futures.items(): + summaries.append(future.result()) - joined_summaries = "\n\n".join(summaries) if summaries else "" + joined_summaries = "\n\n".join(summaries) prompt = ( "Generate a summary based on these summaries of linked notes:\n" "{summaries}\n\n" From 172e5675530f6eb8c17332e408cec5cc0ce2bb6c Mon Sep 17 00:00:00 2001 From: distbit Date: Tue, 6 Jan 2026 09:56:25 +0700 Subject: [PATCH 07/27] Restore detailed edit instructions for spec editing --- src/spec_editing.py | 46 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/src/spec_editing.py b/src/spec_editing.py index 7274323..01e431f 100644 --- a/src/spec_editing.py +++ b/src/spec_editing.py @@ -38,6 +38,50 @@ }, } +INSTRUCTIONS_PROMPT = """# Instructions + +- Integrate the provided notes into the checked-out files. +- Ensure related points are adjacent. +- Break content into relatively atomic bullet points; each bullet should express one idea. +- Use nested bullets when a point is naturally a sub-point of another. +- Make minor grammar edits as needed so ideas read cleanly as bullet points. +- If text to integrate is already well-formatted, punctuated, grammatical and bullet-pointed, avoid altering its wording while integrating/inserting it. +- De-duplicate overlapping points without losing any nuance or detail. +- Keep wording succinct and remove filler words (e.g., "you know", "basically", "essentially", "uh"). +- Add new headings, sub-headings, or parent bullet points for new items, and reuse existing ones where appropriate. +- Refactor existing content as needed to smoothly integrate the new notes. + + +# Rules + +- PRESERVE/DO NOT LEAVE OUT ANY NUANCE, DETAILS, POINTS, CONCLUSIONS, IDEAS, ARGUMENTS, OR QUALIFICATIONS from the notes. +- PRESERVE ALL EXPLANATIONS FROM THE NOTES. +- Do not materially alter meaning. +- If new items do not match existing items in the checked-out files, add them appropriately. +- Preserve questions as questions; do not convert them into statements. +- Do not guess acronym expansions if they are not specified. +- Do not modify tone (e.g., confidence/certainty) or add hedging. +- Do not omit any wikilinks, URLs, diagrams, ASCII art, mathematics, tables, figures, or other non-text content. +- Move each link/URL/etc. to the section where it is most relevant based on its surrounding context and its URL text. + - Do not move links to a separate "resources" or "links" section. +- Do not modify any wikilinks or URLs. + + +# Formatting + +- Use nested markdown headings ("#", "##", "###", "####", etc.) for denoting groups and sub-groups, except if heading text is a [[wikilink]]. + - unless existing content already employs a different convention. +- Use "- " as the bullet prefix (not "* ", "- ", or anything else). + - Use four spaces for each level of bullet-point nesting. + + +# Before finishing: check your work + +- Confirm every item from the provided notes is now represented in the checked-out files without loss of detail. +- Ensure nothing from the original checked-out files was lost. +- If anything is missing, integrate it in appropriately. +""" + @dataclass(frozen=True) class EditInstruction: @@ -88,7 +132,7 @@ def build_edit_prompt( prompt = ( "\n" - f"{instructions}\n" + f"{instructions}\n\n{INSTRUCTIONS_PROMPT.strip()}\n" "\n\n" "\n" f"{chunk_text}\n" From 7506dc356313c430e3ab3d4623ac06ce0ec254d5 Mon Sep 17 00:00:00 2001 From: distbit Date: Tue, 6 Jan 2026 09:59:26 +0700 Subject: [PATCH 08/27] Emphasize multi-level headings in spec edit prompt --- src/spec_editing.py | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/src/spec_editing.py b/src/spec_editing.py index 01e431f..8793c03 100644 --- a/src/spec_editing.py +++ b/src/spec_editing.py @@ -69,7 +69,7 @@ # Formatting -- Use nested markdown headings ("#", "##", "###", "####", etc.) for denoting groups and sub-groups, except if heading text is a [[wikilink]]. +- Use multiple levels of markdown headings ("#", "##", "###", "####", etc.) to express hierarchy, not just top-level headings, except if heading text is a [[wikilink]]. - unless existing content already employs a different convention. - Use "- " as the bullet prefix (not "* ", "- ", or anything else). - Use four spaces for each level of bullet-point nesting. @@ -254,7 +254,9 @@ def _build_whitespace_pattern(text: str, allow_zero: bool) -> re.Pattern[str]: return re.compile(pattern, flags=re.MULTILINE) -def _locate_search_text(body: str, search_text: str) -> tuple[int | None, int | None, str]: +def _locate_search_text( + body: str, search_text: str +) -> tuple[int | None, int | None, str]: attempted_descriptions: List[str] = [] index = body.find(search_text) @@ -331,7 +333,9 @@ def _locate_search_text(body: str, search_text: str) -> tuple[int | None, int | match = matches[0] return match.start(), match.end(), "" - reason = "SEARCH text not found after attempts: " + ", ".join(attempted_descriptions) + reason = "SEARCH text not found after attempts: " + ", ".join( + attempted_descriptions + ) return None, None, reason @@ -343,7 +347,10 @@ def apply_edits( file_contents: Dict[Path, str], edits: List[EditInstruction], ) -> tuple[EditApplication | None, List[EditFailure]]: - updated_contents = {path: _normalize_line_endings(content) for path, content in file_contents.items()} + updated_contents = { + path: _normalize_line_endings(content) + for path, content in file_contents.items() + } failures: List[EditFailure] = [] patch_replacements: List[str] = [] duplicate_texts: List[str] = [] @@ -365,7 +372,9 @@ def apply_edits( duplicate_texts.append(edit.find_text) continue replacement = edit.replace_text or "" - updated_contents[edit.file_path] = _replace_slice(content, start, end, replacement) + updated_contents[edit.file_path] = _replace_slice( + content, start, end, replacement + ) patch_replacements.append(replacement) if failures: @@ -408,9 +417,7 @@ def request_and_apply_edits( except Exception as error: # noqa: BLE001 failed_formatting = str(error) failed_edits = None - logger.warning( - f"Edit response invalid for {attempt_label}: {error}" - ) + logger.warning(f"Edit response invalid for {attempt_label}: {error}") continue failed_formatting = None From 40118a8716c19a9152cbd89226ad6da6d0050da4 Mon Sep 17 00:00:00 2001 From: distbit Date: Tue, 6 Jan 2026 10:00:16 +0700 Subject: [PATCH 09/27] TODO.md SPEC.md spec_editing.py --- TODO.md | 8 +- src/SPEC.md | 334 -------------------------------------------- src/spec_editing.py | 3 +- 3 files changed, 5 insertions(+), 340 deletions(-) diff --git a/TODO.md b/TODO.md index 3d8fa5b..e2b2fbd 100644 --- a/TODO.md +++ b/TODO.md @@ -1,7 +1,7 @@ - ask model to provide snippets from integrate text for each find/replace, to clarify what it intended it to integrate - on fail, only ask the model to provide just that single failed block instead of asking it to provide all blocks again. this is only possible once the model returns what integrated text each block relates - only ask for start and end lines of search block instead of exact text. if there are multiple matches, ask model to provide a sufficient number of of lines at start or end to narrow down options to a single match. ensure that the search block which the verification prompt sees is not affected by this, by populating the SEARCH section of the block given in the verification prompt with the matching text from the file, instead of only including the start and end lines provided by the model -- modify so that it uses tool calling + hierarchical markdown parsing to avoid needing to ever send the entire document to the model, and instead allow the model to find the relevant section(s) (could often be more than one section which should be modified even to integrate a single piece of info) to modify, and then once it has found the sections, it provides the search/replace diffs - - ensure that the model uses arbitrarily nested md headers, to make this approach scalable instead of only e.g. using one level of headings -- move logs outside of src/ -- put group strat into front matter \ No newline at end of file + + + +- make sure prompts mention importance of \ No newline at end of file diff --git a/src/SPEC.md b/src/SPEC.md index 265fafc..e69de29 100644 --- a/src/SPEC.md +++ b/src/SPEC.md @@ -1,334 +0,0 @@ -# Zettelkasten Inbox Integration Script — Specification - -## Overview - -A script that automatically integrates new text from an inbox note into the most relevant existing notes in a zettelkasten-style markdown repository, using LLM-guided exploration of the note graph. - ---- - -## Phase 1: Chunking - -### Input - -* Inbox file containing new text to integrate -* Filenames of 15 randomly sampled files (>300 words each, excluding index notes) from the repository - -### Process - -1. Number each paragraph in the inbox -2. Provide LLM with: - - * The numbered paragraphs - * The 15 sampled filenames (for granularity calibration) -3. LLM returns groups of paragraph numbers representing semantically coherent chunks - -### Constraints - -* Max 600 words per chunk (but never split a single paragraph) -* Paragraphs within a chunk need not be contiguous -* Groups should only combine paragraphs that are clearly same topic/chain of thought - -### Rationale - -* LLM chunking ensures semantic coherence; mechanical chunking conflates proximity with relatedness -* Sampled filenames calibrate the LLM to match existing note granularity (filenames alone convey topic scope without token cost) -* Non-contiguous grouping allows related but separated paragraphs to be processed together - ---- - -## Phase 2: Summary Generation (Preprocessing) - -### Cache Invalidation - -* Store `(file_path, content_hash, summary)` tuples -* Regenerate summary when `hash(current_content) != cached_hash` - -### Summary Generation Rules - -**Standard notes:** - -``` -Generate a 75-100 word summary of this note's content. -Focus on: main topics, key claims, what questions it answers. -``` - -**Index notes** (filename contains "index"): - -``` -Generate a summary based on these summaries of linked notes: -[summaries of all notes linked from this file] -Synthesize into 75-100 words describing what this index covers. -``` - -### Rationale - -* Hash-based invalidation is precise—updates exactly when needed -* Index notes contain mostly links; summarizing their linked content is more informative than summarizing the links themselves - ---- - -## Phase 3: Exploration - -### State Model - -Each file can be in one of three states: - -| State | What LLM sees | How it gets there | -| --------------- | ---------------------------------------------- | --------------------------- | -| **Available** | Filename + summary | Linked from a viewed file | -| **Viewed** | Filename + summary + headings + outgoing links | LLM requested to view it | -| **Checked out** | Full content | LLM selected it for editing | - -### Exploration Flow - -``` -1. Initialize: - - Root file is automatically VIEWED (summary + headings + links shown) - - All files linked from root are AVAILABLE (filename + summary shown) - -2. Exploration loop: - a. LLM sees: chunk + all VIEWED files (summary/headings/links) + AVAILABLE files (filename/summary) - b. LLM returns: list of AVAILABLE files to VIEW (up to 4 per round) - c. For each requested file: - - Change state to VIEWED - - Show summary + headings + outgoing links - - Files it links to become AVAILABLE (if not already viewed) - d. Repeat until LLM signals ready OR limits reached - -3. Checkout: - - LLM selects up to 3 VIEWED files to CHECK OUT - - Full content of checked-out files shown - -4. Edit: - - LLM provides find/replace blocks for checked-out files -``` - -### Limits - -* Max 3 exploration rounds -* Up to 4 files may be VIEWED per round (fewer is fine) -* Max 15 files VIEWED total -* Max 3 files CHECKED OUT - -### Context Management - -* Only summaries (not full content) accumulate during exploration -* Full content only loaded at checkout -* Keeps exploration cheap regardless of depth - -### Rationale - -* Three-state model separates cheap browsing from expensive content loading -* AVAILABLE shows summary so LLM can judge relevance; VIEWED adds structure (headings + links) for navigation decisions -* Summaries + headings provide enough signal for navigation decisions -* Root file treated identically to others; may itself be edited or contain no links - ---- - -## Phase 4: Editing - -### Edit Format - -```json -{ - "edits": [ - { - "file": "filename.md", - "find": "exact text to locate", - "replace": "replacement text", - "is_duplicate": false - }, - { - "file": "other.md", - "find": "text that already covers this", - "is_duplicate": true - } - ] -} -``` - -### Edit Types - -**Standard edit:** `find` + `replace` provided, content is modified - -**Insertion:** `find` contains anchor text, `replace` contains anchor + new content - -```json -{ - "find": "- Link B", - "replace": "- Link B\n- Link C" -} -``` - -**Duplicate marker:** `is_duplicate: true`, only `find` required - -* `find` contains existing text that already covers the chunk content -* No replacement made; serves as visibility into why content wasn't added - -### Validation - -1. For each edit, search for `find` text in specified file using a whitespace-normalized match (treat runs of spaces/tabs/newlines as equivalent, and ignore trivial leading/trailing whitespace differences) to increase match reliability -2. Must match exactly once (zero matches = error, multiple matches = error) -3. On validation failure: return error to LLM, request correction within same conversation - -### Scope - -* Edits can target any CHECKED OUT file -* This includes the root file and index notes - -### Rationale - -* Find/replace is simple and unambiguous; insertion is just a usage pattern, not a separate operation -* Single-match requirement prevents ambiguous edits -* Duplicate flag provides audit trail without cluttering output with identical find/replace pairs -* In-conversation correction leverages existing context rather than restarting - ---- - -## Tool Schema - -### Exploration Tools - -```typescript -// Request to view files (see headings + links in addition to summary) -interface ViewFilesRequest { - action: "view"; - files: string[]; // up to 4, must be AVAILABLE -} - -// Signal ready to check out files for editing -interface CheckoutRequest { - action: "checkout"; - files: string[]; // max 3, must be VIEWED -} -``` - -### Edit Tools - -```typescript -interface Edit { - file: string; - find: string; - replace?: string; // omit if is_duplicate - is_duplicate: boolean; -} - -interface EditRequest { - action: "edit"; - edits: Edit[]; -} -``` - ---- - -## File Annotation Format - -When displaying a VIEWED file: - -```markdown -## [filename.md] - -**Summary:** [75-100 word summary] - -**Headings:** -- # Main Title -- ## Section One -- ## Section Two -- ### Subsection - -**Links to:** -- [[other-note.md]] — [summary of other-note] -- [[another.md]] — [summary of another] -``` - -When displaying an AVAILABLE file: - -```markdown -- [[filename.md]] — [75-100 word summary] -``` - ---- - -## Execution Flow Summary - -``` -┌─────────────────────────────────────────────────────────┐ -│ PREPROCESSING (run periodically) │ -│ - Update stale summaries (hash-based invalidation) │ -│ - Index notes: summarize from linked summaries │ -└─────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────┐ -│ CHUNKING │ -│ - Show LLM: inbox paragraphs + 15 sample filenames │ -│ - LLM returns: paragraph groupings (max 600w each) │ -└─────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────┐ -│ FOR EACH CHUNK: │ -│ │ -│ ┌───────────────────────────────────────────────────┐ │ -│ │ EXPLORE (max 3 rounds, up to 4 files per round, │ │ -│ │ max 15 files viewed total) │ │ -│ │ - AVAILABLE: see filename + summary │ │ -│ │ - VIEWED: see summary + headings + links │ │ -│ │ - Request more files or signal ready │ │ -│ └───────────────────────────────────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌───────────────────────────────────────────────────┐ │ -│ │ CHECKOUT (max 3 files) │ │ -│ │ - Load full content of selected files │ │ -│ └───────────────────────────────────────────────────┘ │ -│ │ │ -│ ▼ │ -│ ┌───────────────────────────────────────────────────┐ │ -│ │ EDIT │ │ -│ │ - LLM provides find/replace blocks │ │ -│ │ - Validate single-match constraint │ │ -│ │ - Apply edits or request correction │ │ -│ └───────────────────────────────────────────────────┘ │ -│ │ -└─────────────────────────────────────────────────────────┘ -``` - ---- - -## Configuration - -```yaml -# Limits -max_exploration_rounds: 3 -max_files_viewed_per_round: 4 -max_files_viewed_total: 15 -max_files_checked_out: 3 -max_chunk_words: 600 -granularity_sample_size: 15 -granularity_sample_min_words: 300 - -# Paths -root_file: "index.md" -inbox_file: "inbox.md" -notes_directory: "./notes" -summary_cache: "./.summary_cache.json" - -# Summary -summary_target_words: 75-100 -index_filename_pattern: "index" -``` - ---- - -## Out of Scope (Deliberate Simplifications) - -| Feature | Reason excluded | -| --------------------------------- | --------------------------------------------------------------- | -| Create new note | Adds complexity; can be added later | -| Explicit insert_after operation | Find/replace pattern sufficient | -| Summary update debouncing | Premature optimization | -| Pre-routing with all summaries | Doesn't scale; exploration achieves same goal | -| Confidence ratings / review queue | Adds friction; start simple | -| Multiple root files / fallbacks | Unnecessary if root is maintained | -| Heading-level routing | Single dimension (files) is simpler than two (files + sections) | \ No newline at end of file diff --git a/src/spec_editing.py b/src/spec_editing.py index 8793c03..423f7c3 100644 --- a/src/spec_editing.py +++ b/src/spec_editing.py @@ -69,8 +69,7 @@ # Formatting -- Use multiple levels of markdown headings ("#", "##", "###", "####", etc.) to express hierarchy, not just top-level headings, except if heading text is a [[wikilink]]. - - unless existing content already employs a different convention. +- Use multiple levels of markdown headings ("#", "##", "###", "####", etc.) to express hierarchy, not just top-level headings - Use "- " as the bullet prefix (not "* ", "- ", or anything else). - Use four spaces for each level of bullet-point nesting. From f36e67507f555bb7450423eb20ce21ea625576c4 Mon Sep 17 00:00:00 2001 From: distbit Date: Sun, 15 Mar 2026 01:45:28 +1100 Subject: [PATCH 10/27] Pin project to Python 3.13 --- .python-version | 1 + CONTEXT_LOG.md | 3 +++ pyproject.toml | 2 +- 3 files changed, 5 insertions(+), 1 deletion(-) create mode 100644 .python-version create mode 100644 CONTEXT_LOG.md diff --git a/.python-version b/.python-version new file mode 100644 index 0000000..24ee5b1 --- /dev/null +++ b/.python-version @@ -0,0 +1 @@ +3.13 diff --git a/CONTEXT_LOG.md b/CONTEXT_LOG.md new file mode 100644 index 0000000..f6c14f6 --- /dev/null +++ b/CONTEXT_LOG.md @@ -0,0 +1,3 @@ +# Runtime Compatibility + +- The dependency lock currently resolves `pydantic-core==2.33.2`, which ships Python 3.13 wheels but falls back to a PyO3 source build on Python 3.14. PyO3 0.24.1 rejects 3.14, so the project needs to stay pinned to Python 3.13 until the dependency set is upgraded to a 3.14-compatible release line. diff --git a/pyproject.toml b/pyproject.toml index efe6e0e..627e559 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -12,7 +12,7 @@ dependencies = [ "requests", "openai>=1.109.1", ] -requires-python = ">=3.13" +requires-python = ">=3.13,<3.14" license = {text = "UNLICENCE"} readme = "README.md" From b62c5c6e3ce6b86a036f10b3b56811269cbed4b0 Mon Sep 17 00:00:00 2001 From: distbit Date: Sun, 15 Mar 2026 05:39:27 +1100 Subject: [PATCH 11/27] TODO.md --- .gitignore | 3 ++- .vscode/settings.json | 6 +++--- TODO.md | 8 +++++++- 3 files changed, 12 insertions(+), 5 deletions(-) diff --git a/.gitignore b/.gitignore index 2876b82..bf10399 100644 --- a/.gitignore +++ b/.gitignore @@ -94,4 +94,5 @@ src/pending_verification_prompts.json src/logs/* -logs/* \ No newline at end of file +logs/* +**/node_modules/ diff --git a/.vscode/settings.json b/.vscode/settings.json index 86dd3d9..01444ba 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -1,6 +1,6 @@ { - "python.terminal.activateEnvironment": true, - "python.terminal.activateEnvInCurrentTerminal": true, "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python", - "python.analysis.extraPaths": ["${workspaceFolder}/src"] + "python.analysis.extraPaths": [ + "${workspaceFolder}/src" + ] } diff --git a/TODO.md b/TODO.md index e2b2fbd..b7ab7a1 100644 --- a/TODO.md +++ b/TODO.md @@ -3,5 +3,11 @@ - only ask for start and end lines of search block instead of exact text. if there are multiple matches, ask model to provide a sufficient number of of lines at start or end to narrow down options to a single match. ensure that the search block which the verification prompt sees is not affected by this, by populating the SEARCH section of the block given in the verification prompt with the matching text from the file, instead of only including the start and end lines provided by the model +- move all files in this repo except integrate_notes.py to a new repo next to this one called zettel_inbox. still make sure to leave a copy of all non-code files in this repo, while also carrying them across. +- add a new tool, in addition to find/replace, for creating a new file. this just takes a file name to create and initial text to add. add to the instructions an explanation that if the model wants to put some of the content from the chunk into a new file and link to it, because it couldn't find anywhere else satisfactory, it should just use the file tool to create the file, and then link to the file using a wikilink [[file_name]]. importantly it must always link to any new files it creates in at least one place (add validation rule for this). enforce that it must write some new text to any new file. -- make sure prompts mention importance of \ No newline at end of file +- figure out how to make my notes work with butter context repo. Add symlinks automatically for all files under infofi into a dir in the butter repo. Check that this supports two way sync +- add guidance re maximum size of file +- add a copy and paste after function. also log all actions +- prevent it from adding text to index gists. Only new files +- use gemini flash \ No newline at end of file From e7ae378484c1a4ab89ca4f05caa9ca7595ea9117 Mon Sep 17 00:00:00 2001 From: distbit Date: Fri, 10 Apr 2026 16:08:14 +0700 Subject: [PATCH 12/27] Switch LLM calls to OpenRouter --- pyproject.toml | 5 ++ src/integrate_notes.py | 53 ++++++++++++++------- src/integrate_notes_spec.py | 4 +- src/spec_config.py | 6 +-- src/spec_llm.py | 84 ++++++++++++++++++++++---------- tests/test_spec_llm.py | 95 +++++++++++++++++++++++++++++++++++++ 6 files changed, 202 insertions(+), 45 deletions(-) create mode 100644 tests/test_spec_llm.py diff --git a/pyproject.toml b/pyproject.toml index 627e559..f3d7be2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -25,3 +25,8 @@ allow-direct-references = true [tool.hatch.build.targets.wheel] packages = ["src"] + +[dependency-groups] +dev = [ + "pytest>=9.0.3", +] diff --git a/src/integrate_notes.py b/src/integrate_notes.py index 398bd9f..f31342f 100644 --- a/src/integrate_notes.py +++ b/src/integrate_notes.py @@ -14,6 +14,7 @@ import shutil import subprocess +from dotenv import load_dotenv from loguru import logger from openai import OpenAI @@ -23,7 +24,9 @@ GROUPING_BLOCK_END = "" DEFAULT_CHUNK_PARAGRAPHS = 30 DEFAULT_CHUNK_MAX_WORDS = 400 -ENV_API_KEY = "OPENAI_API_KEY" +ENV_API_KEY = "OPENROUTER_API_KEY" +OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1" +DEFAULT_MODEL = "openai/gpt-5.4" DEFAULT_MAX_RETRIES = 3 RETRY_INITIAL_DELAY_SECONDS = 2.0 RETRY_BACKOFF_FACTOR = 2.0 @@ -392,13 +395,33 @@ def chunk_paragraphs( return chunks -def create_openai_client() -> OpenAI: +def create_openrouter_client() -> OpenAI: + load_dotenv() api_key = os.getenv(ENV_API_KEY) if not api_key: raise RuntimeError( f"Environment variable {ENV_API_KEY} is required for GPT access." ) - return OpenAI(api_key=api_key) + return OpenAI(api_key=api_key, base_url=OPENROUTER_BASE_URL) + + +def _message_text(message) -> str: + content = message.content + if isinstance(content, str): + return content + if content is None: + return "" + if isinstance(content, list): + parts: List[str] = [] + for item in content: + if isinstance(item, dict): + text = item.get("text") + else: + text = getattr(item, "text", None) + if isinstance(text, str): + parts.append(text) + return "".join(parts) + return str(content) NOTIFY_SEND_PATH = shutil.which("notify-send") @@ -449,11 +472,11 @@ def execute_with_retry( except Exception as error: if attempt >= max_attempts: logger.exception( - f"OpenAI {description} failed after {max_attempts} attempt(s): {error}" + f"OpenRouter {description} failed after {max_attempts} attempt(s): {error}" ) raise logger.warning( - f"OpenAI {description} attempt {attempt} failed: {error}. Retrying in {delay:.1f}s." + f"OpenRouter {description} attempt {attempt} failed: {error}. Retrying in {delay:.1f}s." ) sleep(delay) attempt += 1 @@ -541,12 +564,11 @@ def build_integration_prompt( def request_integration(client: OpenAI, prompt: str, context_label: str) -> str: def perform_request() -> str: - response = client.responses.create( - model="gpt-5.2", - reasoning={"effort": "medium"}, - input=prompt, + response = client.chat.completions.create( + model=DEFAULT_MODEL, + messages=[{"role": "user", "content": prompt}], ) - output_text = response.output_text + output_text = _message_text(response.choices[0].message) if not output_text.strip(): raise RuntimeError("Received empty response from GPT integration call.") patch_text = extract_patch_text_from_response(output_text) @@ -1310,12 +1332,11 @@ def build_verification_prompt( def request_verification(client: OpenAI, prompt: str, context_label: str) -> str: def perform_request() -> str: - response = client.responses.create( - model="gpt-5.2", - reasoning={"effort": "medium"}, - input=prompt, + response = client.chat.completions.create( + model=DEFAULT_MODEL, + messages=[{"role": "user", "content": prompt}], ) - output_text = response.output_text + output_text = _message_text(response.choices[0].message) if not output_text.strip(): raise RuntimeError("Received empty response from GPT verification call.") return output_text.strip() @@ -1410,7 +1431,7 @@ def integrate_notes( ) commit_and_push_original(source_path) scratchpad_paragraphs = normalize_paragraphs(source_scratchpad) - client = create_openai_client() + client = create_openrouter_client() verification_manager = ( None if disable_verification else VerificationManager(client, source_path) ) diff --git a/src/integrate_notes_spec.py b/src/integrate_notes_spec.py index 0bbebc7..00dab3a 100644 --- a/src/integrate_notes_spec.py +++ b/src/integrate_notes_spec.py @@ -18,7 +18,7 @@ ) from spec_editing import request_and_apply_edits from spec_exploration import explore_until_checkout -from spec_llm import create_openai_client +from spec_llm import create_openrouter_client from spec_logging import configure_logging from spec_markdown import ( build_document, @@ -124,7 +124,7 @@ def integrate_notes_spec(source_path: Path, disable_verification: bool) -> Path: scratchpad_paragraphs = normalize_paragraphs(source_scratchpad) repo = NoteRepository(source_path, source_body, source_path.parent) - client = create_openai_client() + client = create_openrouter_client() summaries = SummaryService(repo, client, config) verification_manager = ( None if disable_verification else VerificationManager(client, source_path) diff --git a/src/spec_config.py b/src/spec_config.py index b9af278..0e206fd 100644 --- a/src/spec_config.py +++ b/src/spec_config.py @@ -6,10 +6,10 @@ from typing import Any SCRATCHPAD_HEADING = "# -- SCRATCHPAD" -ENV_API_KEY = "OPENAI_API_KEY" +ENV_API_KEY = "OPENROUTER_API_KEY" +OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1" -DEFAULT_MODEL = "gpt-5.2" -DEFAULT_REASONING = {"effort": "medium"} +DEFAULT_MODEL = "openai/gpt-5.4" DEFAULT_MAX_RETRIES = 3 RETRY_INITIAL_DELAY_SECONDS = 2.0 diff --git a/src/spec_llm.py b/src/spec_llm.py index 40cb049..b79029c 100644 --- a/src/spec_llm.py +++ b/src/spec_llm.py @@ -2,30 +2,71 @@ import json import os +from dataclasses import dataclass from time import sleep -from typing import Iterable +from typing import Any, Iterable +from dotenv import load_dotenv from loguru import logger from openai import OpenAI -from openai.types.responses import ResponseFunctionToolCall from spec_config import ( DEFAULT_MAX_RETRIES, DEFAULT_MODEL, - DEFAULT_REASONING, ENV_API_KEY, + OPENROUTER_BASE_URL, RETRY_BACKOFF_FACTOR, RETRY_INITIAL_DELAY_SECONDS, ) -def create_openai_client() -> OpenAI: +@dataclass(frozen=True) +class ToolCall: + name: str + arguments: str | None + + +def create_openrouter_client() -> OpenAI: + load_dotenv() api_key = os.getenv(ENV_API_KEY) if not api_key: raise RuntimeError( f"Environment variable {ENV_API_KEY} is required for GPT access." ) - return OpenAI(api_key=api_key) + return OpenAI(api_key=api_key, base_url=OPENROUTER_BASE_URL) + + +def _as_chat_tool(tool: dict[str, Any]) -> dict[str, Any]: + if tool.get("type") != "function": + raise ValueError("Only function tools are supported.") + return { + "type": "function", + "function": { + "name": tool["name"], + "description": tool["description"], + "parameters": tool["parameters"], + "strict": tool.get("strict", False), + }, + } + + +def _message_text(message) -> str: + content = message.content + if isinstance(content, str): + return content + if content is None: + return "" + if isinstance(content, list): + parts = [] + for item in content: + if isinstance(item, dict): + text = item.get("text") + else: + text = getattr(item, "text", None) + if isinstance(text, str): + parts.append(text) + return "".join(parts) + return str(content) def execute_with_retry( @@ -43,11 +84,11 @@ def execute_with_retry( except Exception as error: if attempt >= max_attempts: logger.exception( - f"OpenAI {description} failed after {max_attempts} attempt(s): {error}" + f"OpenRouter {description} failed after {max_attempts} attempt(s): {error}" ) raise logger.warning( - f"OpenAI {description} attempt {attempt} failed: {error}. Retrying in {delay:.1f}s." + f"OpenRouter {description} attempt {attempt} failed: {error}. Retrying in {delay:.1f}s." ) sleep(delay) attempt += 1 @@ -56,14 +97,11 @@ def execute_with_retry( def request_text(client: OpenAI, prompt: str, context_label: str) -> str: def perform_request() -> str: - response = client.responses.create( + response = client.chat.completions.create( model=DEFAULT_MODEL, - reasoning=DEFAULT_REASONING, - input=prompt, + messages=[{"role": "user", "content": prompt}], ) - if response.error: - raise RuntimeError(f"OpenAI error for {context_label}: {response.error}") - output_text = response.output_text + output_text = _message_text(response.choices[0].message) if not output_text.strip(): raise RuntimeError(f"Received empty response for {context_label}.") return output_text.strip() @@ -73,31 +111,29 @@ def perform_request() -> str: def request_tool_call( client: OpenAI, prompt: str, tools: Iterable[dict], context_label: str -) -> ResponseFunctionToolCall: - def perform_request() -> ResponseFunctionToolCall: - response = client.responses.create( +) -> ToolCall: + def perform_request() -> ToolCall: + response = client.chat.completions.create( model=DEFAULT_MODEL, - reasoning=DEFAULT_REASONING, - input=prompt, - tools=list(tools), + messages=[{"role": "user", "content": prompt}], + tools=[_as_chat_tool(tool) for tool in tools], tool_choice="required", parallel_tool_calls=False, ) - if response.error: - raise RuntimeError(f"OpenAI error for {context_label}: {response.error}") - tool_calls = [item for item in response.output if item.type == "function_call"] + tool_calls = response.choices[0].message.tool_calls or [] if not tool_calls: raise RuntimeError(f"No tool call returned for {context_label}.") if len(tool_calls) > 1: raise RuntimeError( f"Expected a single tool call for {context_label}, got {len(tool_calls)}." ) - return tool_calls[0] + function = tool_calls[0].function + return ToolCall(function.name, function.arguments) return execute_with_retry(perform_request, context_label) -def parse_tool_call_arguments(call: ResponseFunctionToolCall) -> dict: +def parse_tool_call_arguments(call: ToolCall) -> dict: if not call.arguments: raise RuntimeError(f"Tool call {call.name} missing arguments.") try: diff --git a/tests/test_spec_llm.py b/tests/test_spec_llm.py new file mode 100644 index 0000000..4bbbb82 --- /dev/null +++ b/tests/test_spec_llm.py @@ -0,0 +1,95 @@ +from __future__ import annotations + +import sys +from pathlib import Path +from types import SimpleNamespace + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +import spec_llm # noqa: E402 +from spec_config import DEFAULT_MODEL # noqa: E402 + + +class FakeCompletions: + def __init__(self, message): + self.message = message + self.kwargs = None + + def create(self, **kwargs): + self.kwargs = kwargs + return SimpleNamespace(choices=[SimpleNamespace(message=self.message)]) + + +def fake_client(message): + completions = FakeCompletions(message) + client = SimpleNamespace( + chat=SimpleNamespace(completions=completions), + ) + return client, completions + + +def test_request_text_uses_openrouter_chat_completion_shape(): + client, completions = fake_client(SimpleNamespace(content=" done ")) + + result = spec_llm.request_text(client, "prompt", "unit") + + assert result == "done" + assert completions.kwargs == { + "model": DEFAULT_MODEL, + "messages": [{"role": "user", "content": "prompt"}], + } + + +def test_request_tool_call_converts_response_tool_schema_to_chat_tool_schema(): + message = SimpleNamespace( + content=None, + tool_calls=[ + SimpleNamespace( + function=SimpleNamespace( + name="edit_notes", + arguments='{"action":"edit","edits":[]}', + ) + ) + ], + ) + client, completions = fake_client(message) + tool_schema = { + "type": "function", + "name": "edit_notes", + "description": "Edit checked-out notes.", + "strict": True, + "parameters": {"type": "object", "properties": {}}, + } + + tool_call = spec_llm.request_tool_call(client, "prompt", [tool_schema], "unit") + + assert tool_call.name == "edit_notes" + assert tool_call.arguments == '{"action":"edit","edits":[]}' + assert completions.kwargs == { + "model": DEFAULT_MODEL, + "messages": [{"role": "user", "content": "prompt"}], + "tools": [ + { + "type": "function", + "function": { + "name": "edit_notes", + "description": "Edit checked-out notes.", + "parameters": {"type": "object", "properties": {}}, + "strict": True, + }, + } + ], + "tool_choice": "required", + "parallel_tool_calls": False, + } + + +def test_parse_tool_call_arguments_rejects_non_object_payload(): + call = spec_llm.ToolCall("edit_notes", "[]") + + try: + spec_llm.parse_tool_call_arguments(call) + except RuntimeError as error: + assert "must be a JSON object" in str(error) + else: + raise AssertionError("Expected non-object tool arguments to be rejected.") From 46114345f242a09b73c46f3aacfd3f82acc5bc22 Mon Sep 17 00:00:00 2001 From: distbit Date: Tue, 19 May 2026 23:08:25 +0700 Subject: [PATCH 13/27] Resolve note links by slug key --- CONTEXT_LOG.md | 3 +++ src/spec_exploration.py | 14 +++++++------- src/spec_notes.py | 29 +++++++++++++++++++++++++---- 3 files changed, 35 insertions(+), 11 deletions(-) diff --git a/CONTEXT_LOG.md b/CONTEXT_LOG.md index f6c14f6..952b459 100644 --- a/CONTEXT_LOG.md +++ b/CONTEXT_LOG.md @@ -1,3 +1,6 @@ # Runtime Compatibility - The dependency lock currently resolves `pydantic-core==2.33.2`, which ships Python 3.13 wheels but falls back to a PyO3 source build on Python 3.14. PyO3 0.24.1 rejects 3.14, so the project needs to stay pinned to Python 3.13 until the dependency set is upgraded to a 3.14-compatible release line. +## Notes filename slug migration + +- The notes vault uses hyphen-slug filenames while keeping readable wikilinks. Note resolution and exploration file selection should compare by slug-normalized note title rather than exact filename text. diff --git a/src/spec_exploration.py b/src/spec_exploration.py index c8c1c2f..73c15b2 100644 --- a/src/spec_exploration.py +++ b/src/spec_exploration.py @@ -6,7 +6,7 @@ from spec_config import MAX_TOOL_ATTEMPTS, SpecConfig from spec_llm import parse_tool_call_arguments, request_tool_call -from spec_notes import NoteRepository, ViewedNote +from spec_notes import NoteRepository, ViewedNote, note_slug, readable_note_title VIEW_TOOL_SCHEMA = { @@ -53,7 +53,7 @@ class ExplorationError(RuntimeError): def format_viewed_note(note: ViewedNote) -> str: headings = "\n".join(f"- {heading}" for heading in note.headings) links = "\n".join( - f"- [[{path.name}]] — {summary}" for path, summary in note.link_summaries + f"- [[{readable_note_title(path)}]] — {summary}" for path, summary in note.link_summaries ) return ( f"## [{note.path.name}]\n\n" @@ -64,7 +64,7 @@ def format_viewed_note(note: ViewedNote) -> str: def format_available_note(path: Path, summary: str) -> str: - return f"- [[{path.name}]] — {summary}" + return f"- [[{readable_note_title(path)}]] — {summary}" def build_exploration_prompt( @@ -125,7 +125,7 @@ def _dedupe_preserve_order(values: Iterable[str]) -> List[str]: seen: set[str] = set() result: List[str] = [] for value in values: - key = value.lower() + key = note_slug(value) if key in seen: continue seen.add(key) @@ -154,7 +154,7 @@ def _resolve_requested_paths( ) -> List[Path]: resolved: List[Path] = [] for name in names: - key = name.lower() + key = note_slug(name) path = mapping.get(key) if path is None: raise ExplorationError(f"Requested {label} file '{name}' is not available.") @@ -220,7 +220,7 @@ def explore_until_checkout( raise ExplorationError( "Checkout request exceeds max files allowed." ) - view_map = {path.name.lower(): path for path in viewed.keys()} + view_map = {note_slug(path.name): path for path in viewed.keys()} checkout_paths = _resolve_requested_paths( requested, view_map, "viewed" ) @@ -235,7 +235,7 @@ def explore_until_checkout( requested = _parse_file_list(payload, "view") if len(requested) > config.max_files_viewed_per_round: raise ExplorationError("View request exceeds max files allowed.") - available_map = {path.name.lower(): path for path in available.keys()} + available_map = {note_slug(path.name): path for path in available.keys()} requested_paths = _resolve_requested_paths( requested, available_map, "available" ) diff --git a/src/spec_notes.py b/src/spec_notes.py index 6ceb34d..fb187e6 100644 --- a/src/spec_notes.py +++ b/src/spec_notes.py @@ -7,6 +7,29 @@ from spec_markdown import count_words, extract_headings, extract_wikilinks +def note_slug(value: str) -> str: + stem = Path(value.strip()).name + if stem.lower().endswith(".md"): + stem = stem[:-3] + if "#" in stem: + stem = stem.split("#", 1)[0] + + slug_chars: List[str] = [] + previous_was_separator = False + for character in stem.lower(): + if character.isalnum(): + slug_chars.append(character) + previous_was_separator = False + elif not previous_was_separator: + slug_chars.append("-") + previous_was_separator = True + return "".join(slug_chars).strip("-") + + +def readable_note_title(path: Path) -> str: + return path.stem.replace("-", " ") + + @dataclass(frozen=True) class ViewedNote: path: Path @@ -28,16 +51,14 @@ def _build_file_index(self) -> Dict[str, Path]: mapping: Dict[str, Path] = {} for path in self._notes_dir.iterdir(): if path.is_file() and path.suffix.lower() == ".md": - mapping[path.name.lower()] = path + mapping[note_slug(path.name)] = path return mapping def resolve_link(self, link_text: str) -> Optional[Path]: target = link_text.strip() if not target: return None - if not target.lower().endswith(".md"): - target = f"{target}.md" - return self._file_index.get(target.lower()) + return self._file_index.get(note_slug(target)) def get_note_content(self, path: Path) -> str: if path == self._root_path: From a0c71c69ce6b6251aa29ec545083711e2c1f062f Mon Sep 17 00:00:00 2001 From: distbit Date: Wed, 27 May 2026 18:28:33 +0700 Subject: [PATCH 14/27] Use project-scoped OpenRouter key loading --- CONTEXT_LOG.md | 4 ++++ src/integrate_notes.py | 2 +- src/spec_llm.py | 3 ++- 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/CONTEXT_LOG.md b/CONTEXT_LOG.md index 952b459..7f66c68 100644 --- a/CONTEXT_LOG.md +++ b/CONTEXT_LOG.md @@ -1,6 +1,10 @@ # Runtime Compatibility - The dependency lock currently resolves `pydantic-core==2.33.2`, which ships Python 3.13 wheels but falls back to a PyO3 source build on Python 3.14. PyO3 0.24.1 rejects 3.14, so the project needs to stay pinned to Python 3.13 until the dependency set is upgraded to a 3.14-compatible release line. + +# OpenRouter Env Routing + +- OpenRouter clients load the repo `.env` with override enabled so the repo-specific `OPENROUTER_API_KEY` wins over inherited shell values. This prevents global keys from merging Integrate Notes usage with other projects. ## Notes filename slug migration - The notes vault uses hyphen-slug filenames while keeping readable wikilinks. Note resolution and exploration file selection should compare by slug-normalized note title rather than exact filename text. diff --git a/src/integrate_notes.py b/src/integrate_notes.py index f31342f..afb3066 100644 --- a/src/integrate_notes.py +++ b/src/integrate_notes.py @@ -396,7 +396,7 @@ def chunk_paragraphs( def create_openrouter_client() -> OpenAI: - load_dotenv() + load_dotenv(Path(__file__).resolve().parent.parent / ".env", override=True) api_key = os.getenv(ENV_API_KEY) if not api_key: raise RuntimeError( diff --git a/src/spec_llm.py b/src/spec_llm.py index b79029c..13a1869 100644 --- a/src/spec_llm.py +++ b/src/spec_llm.py @@ -17,6 +17,7 @@ OPENROUTER_BASE_URL, RETRY_BACKOFF_FACTOR, RETRY_INITIAL_DELAY_SECONDS, + repo_root, ) @@ -27,7 +28,7 @@ class ToolCall: def create_openrouter_client() -> OpenAI: - load_dotenv() + load_dotenv(repo_root() / ".env", override=True) api_key = os.getenv(ENV_API_KEY) if not api_key: raise RuntimeError( From 3ac1b078d9f9943a95a072f4f2b754be546a149b Mon Sep 17 00:00:00 2001 From: distbit Date: Sun, 31 May 2026 00:13:27 +0700 Subject: [PATCH 15/27] use frontmatter for note grouping --- CONTEXT_LOG.md | 4 + src/integrate_notes.py | 324 ++++++++++++++++------------- tests/test_frontmatter_grouping.py | 109 ++++++++++ 3 files changed, 290 insertions(+), 147 deletions(-) create mode 100644 tests/test_frontmatter_grouping.py diff --git a/CONTEXT_LOG.md b/CONTEXT_LOG.md index 7f66c68..c4c60e1 100644 --- a/CONTEXT_LOG.md +++ b/CONTEXT_LOG.md @@ -8,3 +8,7 @@ ## Notes filename slug migration - The notes vault uses hyphen-slug filenames while keeping readable wikilinks. Note resolution and exploration file selection should compare by slug-normalized note title rather than exact filename text. + +## Note directive frontmatter + +- The grouping directive is now frontmatter (`grouping: |`) rather than body text near the top of notes. Continuous scratchpad integration is opt-in via `organise: continuous`; pending continuous notes must also have `grouping` frontmatter so batch mode does not prompt mid-run. diff --git a/src/integrate_notes.py b/src/integrate_notes.py index afb3066..0202132 100644 --- a/src/integrate_notes.py +++ b/src/integrate_notes.py @@ -19,9 +19,10 @@ from openai import OpenAI SCRATCHPAD_HEADING = "# -- SCRATCHPAD" -GROUPING_PREFIX = "Grouping approach: " -GROUPING_BLOCK_START = "" -GROUPING_BLOCK_END = "" +GROUPING_FIELD = "grouping" +ORGANISE_FIELD = "organise" +CONTINUOUS_ORGANISE_VALUE = "continuous" +DEFAULT_NOTES_ROOT = Path("/home/pimania/notes") DEFAULT_CHUNK_PARAGRAPHS = 30 DEFAULT_CHUNK_MAX_WORDS = 400 ENV_API_KEY = "OPENROUTER_API_KEY" @@ -119,7 +120,17 @@ def parse_arguments() -> argparse.Namespace: parser.add_argument( "--grouping", required=False, - help="Grouping approach to record at the top of the document.", + help="Grouping approach to record in frontmatter.", + ) + parser.add_argument( + "--continuous", + action="store_true", + help="Integrate all notes marked organise: continuous with non-empty scratchpads.", + ) + parser.add_argument( + "--notes-root", + default=str(DEFAULT_NOTES_ROOT), + help="Notes vault root for --continuous scans.", ) parser.add_argument( "--chunk-size", @@ -165,137 +176,109 @@ def split_document_sections(content: str) -> Tuple[str, str]: return body, scratchpad -@dataclass(frozen=True) -class GroupingSection: - text: str - raw: str - format: str +def split_frontmatter(content: str) -> tuple[list[str], str] | None: + lines = content.splitlines() + if not lines or lines[0].strip() != "---": + return None + for index, line in enumerate(lines[1:], start=1): + if line == "---": + body = "\n".join(lines[index + 1 :]) + if content.endswith("\n"): + body += "\n" + return lines[1:index], body + raise ValueError("Frontmatter starts with '---' but has no closing delimiter.") + + +def _is_top_level_field(line: str) -> bool: + return bool(line.strip()) and not line.startswith((" ", "\t", "-")) and ":" in line + + +def read_frontmatter_field(content: str, field_name: str) -> str | None: + frontmatter_parts = split_frontmatter(content) + if frontmatter_parts is None: + return None + + metadata_lines, _ = frontmatter_parts + field_prefix = f"{field_name}:" + for index, line in enumerate(metadata_lines): + if not line.startswith(field_prefix): + continue + raw_value = line.split(":", 1)[1].strip() + if raw_value in {"|", "|-", "|+", ">", ">-", ">+"}: + block_lines: list[str] = [] + for block_line in metadata_lines[index + 1 :]: + if _is_top_level_field(block_line): + break + block_lines.append( + block_line[2:] if block_line.startswith(" ") else block_line + ) + return "\n".join(block_lines).strip("\n") + return raw_value.strip("\"'") + return None + +def _skip_frontmatter_field(lines: list[str], start_index: int) -> int: + line = lines[start_index] + value = line.split(":", 1)[1].strip() + next_index = start_index + 1 + if value in {"|", "|-", "|+", ">", ">-", ">+"}: + while next_index < len(lines) and not _is_top_level_field(lines[next_index]): + next_index += 1 + return next_index -def _find_front_matter_end_index(lines: List[str]) -> int: + +def _without_frontmatter_field(lines: list[str], field_name: str) -> list[str]: + filtered: list[str] = [] index = 0 - total_lines = len(lines) - while index < total_lines and not lines[index].strip(): - index += 1 - if index < total_lines and lines[index].strip() == "---": + field_prefix = f"{field_name}:" + while index < len(lines): + if lines[index].startswith(field_prefix): + index = _skip_frontmatter_field(lines, index) + continue + filtered.append(lines[index]) index += 1 - while index < total_lines and lines[index].strip() != "---": - index += 1 - if index < total_lines: - index += 1 - return index + return filtered -def _extract_grouping_from_line(line: str) -> str | None: - stripped = line.lstrip() - if not stripped.lower().startswith(GROUPING_PREFIX.lower()): - return None - prefix_length = len(GROUPING_PREFIX) - return stripped[prefix_length:] - - -def extract_grouping_section(body: str) -> tuple[GroupingSection | None, str]: - lines = body.splitlines() - if not lines: - return None, body - - start_index = _find_front_matter_end_index(lines) - grouping_index = start_index - while grouping_index < len(lines) and not lines[grouping_index].strip(): - grouping_index += 1 - - if ( - grouping_index < len(lines) - and lines[grouping_index].strip() == GROUPING_BLOCK_START - ): - end_index = grouping_index + 1 - while end_index < len(lines) and lines[end_index].strip() != GROUPING_BLOCK_END: - end_index += 1 - if end_index >= len(lines): - raise ValueError( - f"Grouping approach block is missing the end marker '{GROUPING_BLOCK_END}'." - ) - grouping_lines = lines[grouping_index + 1 : end_index] - raw_block = "\n".join(lines[grouping_index : end_index + 1]) - grouping_text = "\n".join(grouping_lines) - removal_end = end_index + 1 - if removal_end < len(lines) and not lines[removal_end].strip(): - removal_end += 1 - remaining_lines = lines[:grouping_index] + lines[removal_end:] - return GroupingSection( - text=grouping_text, raw=raw_block, format="block" - ), "\n".join(remaining_lines) - - if grouping_index < len(lines): - grouping_text = _extract_grouping_from_line(lines[grouping_index]) - if grouping_text is not None: - raw_line = lines[grouping_index] - removal_end = grouping_index + 1 - if removal_end < len(lines) and not lines[removal_end].strip(): - removal_end += 1 - remaining_lines = lines[:grouping_index] + lines[removal_end:] - return GroupingSection( - text=grouping_text, raw=raw_line, format="legacy" - ), "\n".join(remaining_lines) - - return None, body - - -def _format_grouping_block(grouping_text: str) -> str: - if grouping_text.endswith("\n"): - return f"{GROUPING_BLOCK_START}\n{grouping_text}{GROUPING_BLOCK_END}" - return f"{GROUPING_BLOCK_START}\n{grouping_text}\n{GROUPING_BLOCK_END}" - - -def render_grouping_section( - grouping_text: str, - existing_section: GroupingSection | None, - preserve_existing: bool, -) -> str: - if not grouping_text.strip(): - raise ValueError("Grouping approach cannot be empty.") - - if preserve_existing and existing_section is not None: - return existing_section.raw - - target_format = existing_section.format if existing_section else "block" - if target_format == "legacy": - if "\n" in grouping_text: - raise ValueError( - "Legacy 'Grouping approach:' format does not support multiline content. " - "Remove the legacy line or add a grouping block instead." - ) - return f"{GROUPING_PREFIX}{grouping_text}" - if target_format == "block": - return _format_grouping_block(grouping_text) - raise ValueError(f"Unknown grouping format: {target_format}") +def _render_block_field(field_name: str, value: str) -> list[str]: + stripped_value = value.strip() + if not stripped_value: + raise ValueError(f"{field_name} cannot be empty.") + return [f"{field_name}: |"] + [ + f" {line}" if line else "" for line in stripped_value.splitlines() + ] -def insert_grouping_section(body: str, grouping_section: str | None) -> str: - if not grouping_section: - return body +def set_frontmatter_block_field(content: str, field_name: str, value: str) -> str: + frontmatter_parts = split_frontmatter(content) + if frontmatter_parts is None: + metadata_lines: list[str] = [] + body = content + else: + metadata_lines, body = frontmatter_parts - lines = body.splitlines() - insertion_index = _find_front_matter_end_index(lines) - while insertion_index < len(lines) and not lines[insertion_index].strip(): - insertion_index += 1 + updated_metadata = _without_frontmatter_field(metadata_lines, field_name) + if updated_metadata and updated_metadata[-1].strip(): + updated_metadata.append("") + updated_metadata.extend(_render_block_field(field_name, value)) - grouping_lines = grouping_section.splitlines() - lines[insertion_index:insertion_index] = grouping_lines - next_index = insertion_index + len(grouping_lines) - if next_index < len(lines) and lines[next_index].strip(): - lines.insert(next_index, "") + frontmatter = "\n".join(updated_metadata).rstrip() + body = body.lstrip("\n") + return f"---\n{frontmatter}\n---\n{body}" - return "\n".join(lines) + +def frontmatter_field_equals(content: str, field_name: str, value: str) -> bool: + field_value = read_frontmatter_field(content, field_name) + return field_value is not None and field_value.strip().lower() == value def prompt_for_grouping() -> str: prompt = ( - "Grouping not found. Provide the text that should follow " - f"{GROUPING_PREFIX} at the top of the document.\n" + "Grouping not found. Provide the text for the frontmatter " + f"{GROUPING_FIELD} field.\n" "Enter multiline text and finish with a single line containing only a '.'.\n" "Examples:\n" - '- Grouping approach: Group points according to what problem each idea/proposal/mechanism/concept addresses/are trying to solve, which you will need to figure out yourself based on context. Do not combine multiple goals/problems into one group. Keep goals/problems specific. Ensure groups are mutually exclusive and collectively exhaustive. Avoid overlap between group\'s goals/problems. sub-headings should be per-mechanism/per-solution i.e. according to which "idea"/solution each point relates to.\n' + '- Group points according to what problem each idea/proposal/mechanism/concept addresses/are trying to solve, which you will need to figure out yourself based on context. Do not combine multiple goals/problems into one group. Keep goals/problems specific. Ensure groups are mutually exclusive and collectively exhaustive. Avoid overlap between group\'s goals/problems. sub-headings should be per-mechanism/per-solution i.e. according to which "idea"/solution each point relates to.\n' "- Group points according to what you think the most useful/interesting/relevant groupings are. Ensure similar, related and contradictory points are adjacent.\n" "Your input:\n" ) @@ -1036,11 +1019,8 @@ def integrate_chunk_with_patches( ) -def build_document( - body: str, grouping_section: str | None, remaining_paragraphs: List[str] -) -> str: - body_with_grouping = insert_grouping_section(body, grouping_section) - trimmed_body = body_with_grouping.rstrip() +def build_document(body: str, remaining_paragraphs: List[str]) -> str: + trimmed_body = body.rstrip() document_parts = [trimmed_body, SCRATCHPAD_HEADING] if remaining_paragraphs: scratchpad_text = "\n\n".join(remaining_paragraphs).rstrip() @@ -1415,19 +1395,15 @@ def integrate_notes( ) -> Path: source_content = source_path.read_text(encoding="utf-8") source_body, source_scratchpad = split_document_sections(source_content) - grouping_section, working_body = extract_grouping_section(source_body) - - resolved_grouping = grouping or ( - grouping_section.text if grouping_section else None - ) + resolved_grouping = grouping or read_frontmatter_field(source_body, GROUPING_FIELD) if not resolved_grouping: resolved_grouping = prompt_for_grouping() logger.info("Recorded new grouping approach from user input.") - grouping_section_text = render_grouping_section( - resolved_grouping, - grouping_section, - preserve_existing=grouping is None and grouping_section is not None, + working_body = ( + source_body + if grouping is None and read_frontmatter_field(source_body, GROUPING_FIELD) + else set_frontmatter_block_field(source_body, GROUPING_FIELD, resolved_grouping) ) commit_and_push_original(source_path) scratchpad_paragraphs = normalize_paragraphs(source_scratchpad) @@ -1442,7 +1418,7 @@ def integrate_notes( "No scratchpad notes to integrate; ensuring scratchpad heading remains present." ) source_path.write_text( - build_document(working_body, grouping_section_text, []), + build_document(working_body, []), encoding="utf-8", ) return source_path @@ -1507,9 +1483,7 @@ def integrate_notes( source_path, last_written_remaining ) remaining_paragraphs = refreshed_paragraphs[len(chunk) :] - integrated_document = build_document( - current_body, grouping_section_text, remaining_paragraphs - ) + integrated_document = build_document(current_body, remaining_paragraphs) source_path.write_text(integrated_document, encoding="utf-8") logger.info( f'Chunk {chunks_completed + 1} integration written to "{source_path}".' @@ -1533,21 +1507,77 @@ def integrate_notes( verification_manager.shutdown() +def continuous_organise_paths(notes_root: Path) -> list[Path]: + if not notes_root.exists(): + raise FileNotFoundError(f"Notes root not found: {notes_root}") + if not notes_root.is_dir(): + raise NotADirectoryError(f"Notes root is not a directory: {notes_root}") + + paths: list[Path] = [] + for path in sorted(notes_root.rglob("*.md")): + if any(part.startswith(".") for part in path.relative_to(notes_root).parts): + continue + content = path.read_text(encoding="utf-8") + if not frontmatter_field_equals( + content, ORGANISE_FIELD, CONTINUOUS_ORGANISE_VALUE + ): + continue + _, scratchpad = split_document_sections(content) + if normalize_paragraphs(scratchpad): + if not read_frontmatter_field(content, GROUPING_FIELD): + raise RuntimeError( + f"{path} is marked {ORGANISE_FIELD}: {CONTINUOUS_ORGANISE_VALUE} " + f"but has no {GROUPING_FIELD} frontmatter." + ) + paths.append(path) + return paths + + +def integrate_continuous_notes( + notes_root: Path, + max_paragraphs_per_chunk: int, + max_words_per_chunk: int, + disable_verification: bool, +) -> list[Path]: + source_paths = continuous_organise_paths(notes_root) + for source_path in source_paths: + logger.info(f"Integrating continuously organised note {source_path}.") + integrate_notes( + source_path, + grouping=None, + max_paragraphs_per_chunk=max_paragraphs_per_chunk, + max_words_per_chunk=max_words_per_chunk, + disable_verification=disable_verification, + ) + return source_paths + + def main() -> None: configure_logging() try: args = parse_arguments() - source_path = resolve_source_path(args.source) - integrated_path = integrate_notes( - source_path, - args.grouping, - args.chunk_size, - args.max_chunk_words, - args.disable_verification, - ) - logger.info( - f"Integration completed. Updated document available at {integrated_path}." - ) + if args.continuous: + integrated_paths = integrate_continuous_notes( + Path(args.notes_root).expanduser().resolve(), + args.chunk_size, + args.max_chunk_words, + args.disable_verification, + ) + logger.info( + f"Continuous integration completed for {len(integrated_paths)} note(s)." + ) + else: + source_path = resolve_source_path(args.source) + integrated_path = integrate_notes( + source_path, + args.grouping, + args.chunk_size, + args.max_chunk_words, + args.disable_verification, + ) + logger.info( + f"Integration completed. Updated document available at {integrated_path}." + ) except Exception as error: logger.exception(f"Integration failed: {error}") sys.exit(1) diff --git a/tests/test_frontmatter_grouping.py b/tests/test_frontmatter_grouping.py new file mode 100644 index 0000000..7a68a6e --- /dev/null +++ b/tests/test_frontmatter_grouping.py @@ -0,0 +1,109 @@ +from pathlib import Path +import sys + +import pytest + + +REPO_ROOT = Path(__file__).resolve().parents[1] +SRC_DIR = REPO_ROOT / "src" + +if str(SRC_DIR) not in sys.path: + sys.path.insert(0, str(SRC_DIR)) + +import integrate_notes # noqa: E402 + + +def test_set_frontmatter_block_field_preserves_body_and_replaces_field() -> None: + content = ( + "---\n" + "title: Existing\n" + "grouping: old\n" + "---\n" + "# Body\n\n" + "Text\n" + ) + + updated = integrate_notes.set_frontmatter_block_field( + content, + "grouping", + "Group by topic\nThen by mechanism", + ) + + assert integrate_notes.read_frontmatter_field(updated, "title") == "Existing" + assert integrate_notes.read_frontmatter_field(updated, "grouping") == ( + "Group by topic\nThen by mechanism" + ) + assert updated.endswith("# Body\n\nText\n") + assert "grouping: old" not in updated + + +def test_frontmatter_block_field_can_contain_separator_lines() -> None: + updated = integrate_notes.set_frontmatter_block_field( + "# Body\n", + "grouping", + "First group\n---\nSecond group", + ) + + assert integrate_notes.read_frontmatter_field(updated, "grouping") == ( + "First group\n---\nSecond group" + ) + assert updated.endswith("# Body\n") + + +def test_set_frontmatter_block_field_creates_frontmatter_when_missing() -> None: + updated = integrate_notes.set_frontmatter_block_field( + "# Body\n", + "grouping", + "Group by question", + ) + + assert updated.startswith("---\ngrouping: |\n Group by question\n---\n") + assert updated.endswith("# Body\n") + + +def test_continuous_organise_paths_selects_non_empty_scratchpads( + tmp_path: Path, +) -> None: + ready_note = tmp_path / "ready.md" + ready_note.write_text( + "---\n" + "organise: continuous\n" + "grouping: |\n" + " Group by topic\n" + "---\n" + "# Body\n\n" + "# -- SCRATCHPAD\n\n" + "new point\n", + encoding="utf-8", + ) + empty_scratchpad = tmp_path / "empty.md" + empty_scratchpad.write_text( + "---\n" + "organise: continuous\n" + "grouping: topic\n" + "---\n" + "# Body\n\n" + "# -- SCRATCHPAD\n", + encoding="utf-8", + ) + hidden_dir = tmp_path / ".hidden" + hidden_dir.mkdir() + (hidden_dir / "hidden.md").write_text( + "---\norganise: continuous\ngrouping: topic\n---\n# -- SCRATCHPAD\n\nx\n", + encoding="utf-8", + ) + + assert integrate_notes.continuous_organise_paths(tmp_path) == [ready_note] + + +def test_continuous_organise_paths_requires_grouping_for_pending_notes( + tmp_path: Path, +) -> None: + note = tmp_path / "missing-grouping.md" + note.write_text( + "---\norganise: continuous\n---\n# Body\n\n# -- SCRATCHPAD\n\nnew point\n", + encoding="utf-8", + ) + + with pytest.raises(RuntimeError, match="has no grouping frontmatter"): + integrate_notes.continuous_organise_paths(tmp_path) From 173fb6be5809f6af5f5224c0685eea6aad8b35d3 Mon Sep 17 00:00:00 2001 From: distbit Date: Tue, 2 Jun 2026 17:52:15 +0700 Subject: [PATCH 16/27] Document continuous note organisation timer --- CONTEXT_LOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CONTEXT_LOG.md b/CONTEXT_LOG.md index c4c60e1..1b90191 100644 --- a/CONTEXT_LOG.md +++ b/CONTEXT_LOG.md @@ -12,3 +12,4 @@ ## Note directive frontmatter - The grouping directive is now frontmatter (`grouping: |`) rather than body text near the top of notes. Continuous scratchpad integration is opt-in via `organise: continuous`; pending continuous notes must also have `grouping` frontmatter so batch mode does not prompt mid-run. +- `continuous-note-organisation.timer` runs `src/integrate_notes.py --continuous --notes-root /home/pimania/notes` daily at 09:00 as a systemd user timer. The timer unit is stored outside this repo under `~/.config/systemd/user/`. From dca1e9c6b5276dca2933152ccf633fba56e36450 Mon Sep 17 00:00:00 2001 From: distbit Date: Tue, 2 Jun 2026 17:54:31 +0700 Subject: [PATCH 17/27] Default missing continuous grouping --- CONTEXT_LOG.md | 1 + src/integrate_notes.py | 17 +++++++++++------ tests/test_frontmatter_grouping.py | 15 +++++++++------ 3 files changed, 21 insertions(+), 12 deletions(-) diff --git a/CONTEXT_LOG.md b/CONTEXT_LOG.md index 1b90191..6624b19 100644 --- a/CONTEXT_LOG.md +++ b/CONTEXT_LOG.md @@ -13,3 +13,4 @@ - The grouping directive is now frontmatter (`grouping: |`) rather than body text near the top of notes. Continuous scratchpad integration is opt-in via `organise: continuous`; pending continuous notes must also have `grouping` frontmatter so batch mode does not prompt mid-run. - `continuous-note-organisation.timer` runs `src/integrate_notes.py --continuous --notes-root /home/pimania/notes` daily at 09:00 as a systemd user timer. The timer unit is stored outside this repo under `~/.config/systemd/user/`. +- Continuous mode now writes a default `grouping: |` frontmatter value, with a warning log, when a note is marked `organise: continuous` but has no grouping. This keeps scheduled runs non-interactive while making the default explicit in the note. diff --git a/src/integrate_notes.py b/src/integrate_notes.py index 0202132..9161aae 100644 --- a/src/integrate_notes.py +++ b/src/integrate_notes.py @@ -22,6 +22,7 @@ GROUPING_FIELD = "grouping" ORGANISE_FIELD = "organise" CONTINUOUS_ORGANISE_VALUE = "continuous" +DEFAULT_GROUPING = "Group points according to what you think the most useful/interesting/relevant groupings are. Ensure similar, related and contradictory points are adjacent." DEFAULT_NOTES_ROOT = Path("/home/pimania/notes") DEFAULT_CHUNK_PARAGRAPHS = 30 DEFAULT_CHUNK_MAX_WORDS = 400 @@ -294,7 +295,7 @@ def prompt_for_grouping() -> str: lines.append(line) grouping = "\n".join(lines) if not grouping.strip(): - grouping = "Group points according to what you think the most useful/interesting/relevant groupings are. Ensure similar, related and contradictory points are adjacent." + grouping = DEFAULT_GROUPING return grouping @@ -1522,13 +1523,17 @@ def continuous_organise_paths(notes_root: Path) -> list[Path]: content, ORGANISE_FIELD, CONTINUOUS_ORGANISE_VALUE ): continue + if not read_frontmatter_field(content, GROUPING_FIELD): + content = set_frontmatter_block_field( + content, GROUPING_FIELD, DEFAULT_GROUPING + ) + path.write_text(content, encoding="utf-8") + logger.warning( + f"{path} is marked {ORGANISE_FIELD}: {CONTINUOUS_ORGANISE_VALUE} " + f"but had no {GROUPING_FIELD} frontmatter; added default grouping." + ) _, scratchpad = split_document_sections(content) if normalize_paragraphs(scratchpad): - if not read_frontmatter_field(content, GROUPING_FIELD): - raise RuntimeError( - f"{path} is marked {ORGANISE_FIELD}: {CONTINUOUS_ORGANISE_VALUE} " - f"but has no {GROUPING_FIELD} frontmatter." - ) paths.append(path) return paths diff --git a/tests/test_frontmatter_grouping.py b/tests/test_frontmatter_grouping.py index 7a68a6e..07b8c6a 100644 --- a/tests/test_frontmatter_grouping.py +++ b/tests/test_frontmatter_grouping.py @@ -1,7 +1,5 @@ -from pathlib import Path import sys - -import pytest +from pathlib import Path REPO_ROOT = Path(__file__).resolve().parents[1] @@ -96,7 +94,7 @@ def test_continuous_organise_paths_selects_non_empty_scratchpads( assert integrate_notes.continuous_organise_paths(tmp_path) == [ready_note] -def test_continuous_organise_paths_requires_grouping_for_pending_notes( +def test_continuous_organise_paths_adds_default_grouping_for_pending_notes( tmp_path: Path, ) -> None: note = tmp_path / "missing-grouping.md" @@ -105,5 +103,10 @@ def test_continuous_organise_paths_requires_grouping_for_pending_notes( encoding="utf-8", ) - with pytest.raises(RuntimeError, match="has no grouping frontmatter"): - integrate_notes.continuous_organise_paths(tmp_path) + assert integrate_notes.continuous_organise_paths(tmp_path) == [note] + + updated = note.read_text(encoding="utf-8") + assert integrate_notes.read_frontmatter_field(updated, "grouping") == ( + integrate_notes.DEFAULT_GROUPING + ) + assert "# -- SCRATCHPAD\n\nnew point\n" in updated From 807b2bfeb1b39f5cefce226d1f68958f20c5b6ba Mon Sep 17 00:00:00 2001 From: distbit Date: Sun, 7 Jun 2026 22:52:35 +0700 Subject: [PATCH 18/27] docs: record continuous timer path dependency --- CONTEXT_LOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CONTEXT_LOG.md b/CONTEXT_LOG.md index 6624b19..c0b5782 100644 --- a/CONTEXT_LOG.md +++ b/CONTEXT_LOG.md @@ -13,4 +13,5 @@ - The grouping directive is now frontmatter (`grouping: |`) rather than body text near the top of notes. Continuous scratchpad integration is opt-in via `organise: continuous`; pending continuous notes must also have `grouping` frontmatter so batch mode does not prompt mid-run. - `continuous-note-organisation.timer` runs `src/integrate_notes.py --continuous --notes-root /home/pimania/notes` daily at 09:00 as a systemd user timer. The timer unit is stored outside this repo under `~/.config/systemd/user/`. +- The timer depends on Git hooks in `/home/pimania/notes` being able to find `git-lfs`. On 2026-06-07, continuous integration failed at the pre-integration `git push` because user systemd's PATH omitted Homebrew (`/home/linuxbrew/.linuxbrew/bin`), where `git-lfs` is installed. The persistent fix is `~/.config/environment.d/10-user-path.conf`; the running user manager was also updated with `systemctl --user set-environment`. - Continuous mode now writes a default `grouping: |` frontmatter value, with a warning log, when a note is marked `organise: continuous` but has no grouping. This keeps scheduled runs non-interactive while making the default explicit in the note. From a1a10e12ff9540526136aaec7790d297942d0f5d Mon Sep 17 00:00:00 2001 From: distbit Date: Sun, 7 Jun 2026 23:17:21 +0700 Subject: [PATCH 19/27] fix: bound OpenRouter request duration --- src/integrate_notes.py | 7 +++- src/spec_config.py | 1 + src/spec_llm.py | 7 +++- tests/test_openrouter_client_timeout.py | 46 +++++++++++++++++++++++++ 4 files changed, 59 insertions(+), 2 deletions(-) create mode 100644 tests/test_openrouter_client_timeout.py diff --git a/src/integrate_notes.py b/src/integrate_notes.py index 9161aae..a9a6eac 100644 --- a/src/integrate_notes.py +++ b/src/integrate_notes.py @@ -32,6 +32,7 @@ DEFAULT_MAX_RETRIES = 3 RETRY_INITIAL_DELAY_SECONDS = 2.0 RETRY_BACKOFF_FACTOR = 2.0 +OPENROUTER_REQUEST_TIMEOUT_SECONDS = 120.0 PENDING_VERIFICATION_PROMPTS_PATH = ( Path(__file__).resolve().parent / "pending_verification_prompts.json" ) @@ -386,7 +387,11 @@ def create_openrouter_client() -> OpenAI: raise RuntimeError( f"Environment variable {ENV_API_KEY} is required for GPT access." ) - return OpenAI(api_key=api_key, base_url=OPENROUTER_BASE_URL) + return OpenAI( + api_key=api_key, + base_url=OPENROUTER_BASE_URL, + timeout=OPENROUTER_REQUEST_TIMEOUT_SECONDS, + ) def _message_text(message) -> str: diff --git a/src/spec_config.py b/src/spec_config.py index 0e206fd..821aeb8 100644 --- a/src/spec_config.py +++ b/src/spec_config.py @@ -14,6 +14,7 @@ DEFAULT_MAX_RETRIES = 3 RETRY_INITIAL_DELAY_SECONDS = 2.0 RETRY_BACKOFF_FACTOR = 2.0 +OPENROUTER_REQUEST_TIMEOUT_SECONDS = 120.0 MAX_PATCH_ATTEMPTS = 3 MAX_TOOL_ATTEMPTS = 3 diff --git a/src/spec_llm.py b/src/spec_llm.py index 13a1869..725f49b 100644 --- a/src/spec_llm.py +++ b/src/spec_llm.py @@ -15,6 +15,7 @@ DEFAULT_MODEL, ENV_API_KEY, OPENROUTER_BASE_URL, + OPENROUTER_REQUEST_TIMEOUT_SECONDS, RETRY_BACKOFF_FACTOR, RETRY_INITIAL_DELAY_SECONDS, repo_root, @@ -34,7 +35,11 @@ def create_openrouter_client() -> OpenAI: raise RuntimeError( f"Environment variable {ENV_API_KEY} is required for GPT access." ) - return OpenAI(api_key=api_key, base_url=OPENROUTER_BASE_URL) + return OpenAI( + api_key=api_key, + base_url=OPENROUTER_BASE_URL, + timeout=OPENROUTER_REQUEST_TIMEOUT_SECONDS, + ) def _as_chat_tool(tool: dict[str, Any]) -> dict[str, Any]: diff --git a/tests/test_openrouter_client_timeout.py b/tests/test_openrouter_client_timeout.py new file mode 100644 index 0000000..221a3ef --- /dev/null +++ b/tests/test_openrouter_client_timeout.py @@ -0,0 +1,46 @@ +import sys +from pathlib import Path + + +SRC_DIR = Path(__file__).resolve().parents[1] / "src" +if str(SRC_DIR) not in sys.path: + sys.path.insert(0, str(SRC_DIR)) + +import integrate_notes # noqa: E402 +import spec_llm # noqa: E402 +from spec_config import OPENROUTER_REQUEST_TIMEOUT_SECONDS # noqa: E402 + + +def test_integrate_notes_openrouter_client_uses_bounded_timeout(monkeypatch) -> None: + captured_kwargs = {} + client = object() + + def openai_client(**kwargs): + captured_kwargs.update(kwargs) + return client + + monkeypatch.setattr(integrate_notes, "OpenAI", openai_client) + monkeypatch.setattr(integrate_notes, "load_dotenv", lambda *_, **__: None) + monkeypatch.setenv(integrate_notes.ENV_API_KEY, "test-key") + + assert integrate_notes.create_openrouter_client() is client + assert ( + captured_kwargs["timeout"] + == integrate_notes.OPENROUTER_REQUEST_TIMEOUT_SECONDS + ) + + +def test_spec_openrouter_client_uses_bounded_timeout(monkeypatch) -> None: + captured_kwargs = {} + client = object() + + def openai_client(**kwargs): + captured_kwargs.update(kwargs) + return client + + monkeypatch.setattr(spec_llm, "OpenAI", openai_client) + monkeypatch.setattr(spec_llm, "load_dotenv", lambda *_, **__: None) + monkeypatch.setenv(spec_llm.ENV_API_KEY, "test-key") + + assert spec_llm.create_openrouter_client() is client + assert captured_kwargs["timeout"] == OPENROUTER_REQUEST_TIMEOUT_SECONDS From 32c4d3e0741e87f6c68a8df808c9c74c0ba3d231 Mon Sep 17 00:00:00 2001 From: distbit Date: Sun, 7 Jun 2026 23:22:13 +0700 Subject: [PATCH 20/27] fix: keep OpenRouter retries explicit --- src/integrate_notes.py | 2 ++ src/spec_config.py | 1 + src/spec_llm.py | 2 ++ tests/test_openrouter_client_timeout.py | 7 ++++++- 4 files changed, 11 insertions(+), 1 deletion(-) diff --git a/src/integrate_notes.py b/src/integrate_notes.py index a9a6eac..81cc704 100644 --- a/src/integrate_notes.py +++ b/src/integrate_notes.py @@ -33,6 +33,7 @@ RETRY_INITIAL_DELAY_SECONDS = 2.0 RETRY_BACKOFF_FACTOR = 2.0 OPENROUTER_REQUEST_TIMEOUT_SECONDS = 120.0 +OPENROUTER_SDK_MAX_RETRIES = 0 PENDING_VERIFICATION_PROMPTS_PATH = ( Path(__file__).resolve().parent / "pending_verification_prompts.json" ) @@ -391,6 +392,7 @@ def create_openrouter_client() -> OpenAI: api_key=api_key, base_url=OPENROUTER_BASE_URL, timeout=OPENROUTER_REQUEST_TIMEOUT_SECONDS, + max_retries=OPENROUTER_SDK_MAX_RETRIES, ) diff --git a/src/spec_config.py b/src/spec_config.py index 821aeb8..79d004d 100644 --- a/src/spec_config.py +++ b/src/spec_config.py @@ -15,6 +15,7 @@ RETRY_INITIAL_DELAY_SECONDS = 2.0 RETRY_BACKOFF_FACTOR = 2.0 OPENROUTER_REQUEST_TIMEOUT_SECONDS = 120.0 +OPENROUTER_SDK_MAX_RETRIES = 0 MAX_PATCH_ATTEMPTS = 3 MAX_TOOL_ATTEMPTS = 3 diff --git a/src/spec_llm.py b/src/spec_llm.py index 725f49b..00c79de 100644 --- a/src/spec_llm.py +++ b/src/spec_llm.py @@ -16,6 +16,7 @@ ENV_API_KEY, OPENROUTER_BASE_URL, OPENROUTER_REQUEST_TIMEOUT_SECONDS, + OPENROUTER_SDK_MAX_RETRIES, RETRY_BACKOFF_FACTOR, RETRY_INITIAL_DELAY_SECONDS, repo_root, @@ -39,6 +40,7 @@ def create_openrouter_client() -> OpenAI: api_key=api_key, base_url=OPENROUTER_BASE_URL, timeout=OPENROUTER_REQUEST_TIMEOUT_SECONDS, + max_retries=OPENROUTER_SDK_MAX_RETRIES, ) diff --git a/tests/test_openrouter_client_timeout.py b/tests/test_openrouter_client_timeout.py index 221a3ef..8e0871f 100644 --- a/tests/test_openrouter_client_timeout.py +++ b/tests/test_openrouter_client_timeout.py @@ -8,7 +8,10 @@ import integrate_notes # noqa: E402 import spec_llm # noqa: E402 -from spec_config import OPENROUTER_REQUEST_TIMEOUT_SECONDS # noqa: E402 +from spec_config import ( # noqa: E402 + OPENROUTER_REQUEST_TIMEOUT_SECONDS, + OPENROUTER_SDK_MAX_RETRIES, +) def test_integrate_notes_openrouter_client_uses_bounded_timeout(monkeypatch) -> None: @@ -28,6 +31,7 @@ def openai_client(**kwargs): captured_kwargs["timeout"] == integrate_notes.OPENROUTER_REQUEST_TIMEOUT_SECONDS ) + assert captured_kwargs["max_retries"] == integrate_notes.OPENROUTER_SDK_MAX_RETRIES def test_spec_openrouter_client_uses_bounded_timeout(monkeypatch) -> None: @@ -44,3 +48,4 @@ def openai_client(**kwargs): assert spec_llm.create_openrouter_client() is client assert captured_kwargs["timeout"] == OPENROUTER_REQUEST_TIMEOUT_SECONDS + assert captured_kwargs["max_retries"] == OPENROUTER_SDK_MAX_RETRIES From dfb5d168ffd2081a5a56f7c4f9428151371f85ac Mon Sep 17 00:00:00 2001 From: distbit Date: Sun, 7 Jun 2026 23:28:26 +0700 Subject: [PATCH 21/27] fix: pass OpenRouter timeout at request sites --- src/integrate_notes.py | 2 ++ src/spec_llm.py | 2 ++ tests/test_openrouter_client_timeout.py | 29 +++++++++++++++++++++++++ tests/test_spec_llm.py | 3 +++ 4 files changed, 36 insertions(+) diff --git a/src/integrate_notes.py b/src/integrate_notes.py index 81cc704..b719b6c 100644 --- a/src/integrate_notes.py +++ b/src/integrate_notes.py @@ -558,6 +558,7 @@ def perform_request() -> str: response = client.chat.completions.create( model=DEFAULT_MODEL, messages=[{"role": "user", "content": prompt}], + timeout=OPENROUTER_REQUEST_TIMEOUT_SECONDS, ) output_text = _message_text(response.choices[0].message) if not output_text.strip(): @@ -1323,6 +1324,7 @@ def perform_request() -> str: response = client.chat.completions.create( model=DEFAULT_MODEL, messages=[{"role": "user", "content": prompt}], + timeout=OPENROUTER_REQUEST_TIMEOUT_SECONDS, ) output_text = _message_text(response.choices[0].message) if not output_text.strip(): diff --git a/src/spec_llm.py b/src/spec_llm.py index 00c79de..c372e3b 100644 --- a/src/spec_llm.py +++ b/src/spec_llm.py @@ -108,6 +108,7 @@ def perform_request() -> str: response = client.chat.completions.create( model=DEFAULT_MODEL, messages=[{"role": "user", "content": prompt}], + timeout=OPENROUTER_REQUEST_TIMEOUT_SECONDS, ) output_text = _message_text(response.choices[0].message) if not output_text.strip(): @@ -127,6 +128,7 @@ def perform_request() -> ToolCall: tools=[_as_chat_tool(tool) for tool in tools], tool_choice="required", parallel_tool_calls=False, + timeout=OPENROUTER_REQUEST_TIMEOUT_SECONDS, ) tool_calls = response.choices[0].message.tool_calls or [] if not tool_calls: diff --git a/tests/test_openrouter_client_timeout.py b/tests/test_openrouter_client_timeout.py index 8e0871f..8ba6612 100644 --- a/tests/test_openrouter_client_timeout.py +++ b/tests/test_openrouter_client_timeout.py @@ -1,5 +1,6 @@ import sys from pathlib import Path +from types import SimpleNamespace SRC_DIR = Path(__file__).resolve().parents[1] / "src" @@ -49,3 +50,31 @@ def openai_client(**kwargs): assert spec_llm.create_openrouter_client() is client assert captured_kwargs["timeout"] == OPENROUTER_REQUEST_TIMEOUT_SECONDS assert captured_kwargs["max_retries"] == OPENROUTER_SDK_MAX_RETRIES + + +def test_integration_request_sets_per_call_timeout() -> None: + captured_kwargs = {} + patch_response = ( + f"{integrate_notes.PATCH_BLOCK_START}\n" + "# Body\n" + f"{integrate_notes.PATCH_BLOCK_DIVIDER}\n" + "# Body\n" + f"{integrate_notes.PATCH_BLOCK_END}" + ) + + class Completions: + def create(self, **kwargs): + captured_kwargs.update(kwargs) + return SimpleNamespace( + choices=[ + SimpleNamespace(message=SimpleNamespace(content=patch_response)) + ] + ) + + client = SimpleNamespace(chat=SimpleNamespace(completions=Completions())) + + assert integrate_notes.request_integration(client, "prompt", "unit") + assert ( + captured_kwargs["timeout"] + == integrate_notes.OPENROUTER_REQUEST_TIMEOUT_SECONDS + ) diff --git a/tests/test_spec_llm.py b/tests/test_spec_llm.py index 4bbbb82..2d94f66 100644 --- a/tests/test_spec_llm.py +++ b/tests/test_spec_llm.py @@ -8,6 +8,7 @@ import spec_llm # noqa: E402 from spec_config import DEFAULT_MODEL # noqa: E402 +from spec_config import OPENROUTER_REQUEST_TIMEOUT_SECONDS # noqa: E402 class FakeCompletions: @@ -37,6 +38,7 @@ def test_request_text_uses_openrouter_chat_completion_shape(): assert completions.kwargs == { "model": DEFAULT_MODEL, "messages": [{"role": "user", "content": "prompt"}], + "timeout": OPENROUTER_REQUEST_TIMEOUT_SECONDS, } @@ -81,6 +83,7 @@ def test_request_tool_call_converts_response_tool_schema_to_chat_tool_schema(): ], "tool_choice": "required", "parallel_tool_calls": False, + "timeout": OPENROUTER_REQUEST_TIMEOUT_SECONDS, } From 7bbc1e11c2acbfbf14a0ce41a74390f8b0ceb0d3 Mon Sep 17 00:00:00 2001 From: distbit Date: Sun, 7 Jun 2026 23:39:33 +0700 Subject: [PATCH 22/27] docs: record minimax patch formatting failure --- CONTEXT_LOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CONTEXT_LOG.md b/CONTEXT_LOG.md index c0b5782..449bbd8 100644 --- a/CONTEXT_LOG.md +++ b/CONTEXT_LOG.md @@ -14,4 +14,5 @@ - The grouping directive is now frontmatter (`grouping: |`) rather than body text near the top of notes. Continuous scratchpad integration is opt-in via `organise: continuous`; pending continuous notes must also have `grouping` frontmatter so batch mode does not prompt mid-run. - `continuous-note-organisation.timer` runs `src/integrate_notes.py --continuous --notes-root /home/pimania/notes` daily at 09:00 as a systemd user timer. The timer unit is stored outside this repo under `~/.config/systemd/user/`. - The timer depends on Git hooks in `/home/pimania/notes` being able to find `git-lfs`. On 2026-06-07, continuous integration failed at the pre-integration `git push` because user systemd's PATH omitted Homebrew (`/home/linuxbrew/.linuxbrew/bin`), where `git-lfs` is installed. The persistent fix is `~/.config/environment.d/10-user-path.conf`; the running user manager was also updated with `systemctl --user set-environment`. +- On 2026-06-07, manual reruns with an uncommitted `DEFAULT_MODEL = "minimax/minimax-m3"` change reached the LLM step but failed chunk 1 because the model repeatedly emitted malformed patch blocks, including extra `SEARCH` text inside the search span and a combined `<<<<<<< SEARCH DUPLICATE` marker. The script correctly refused to write these patches. - Continuous mode now writes a default `grouping: |` frontmatter value, with a warning log, when a note is marked `organise: continuous` but has no grouping. This keeps scheduled runs non-interactive while making the default explicit in the note. From 43bfbf893c81cda3084646395a4b355c4925cd67 Mon Sep 17 00:00:00 2001 From: distbit Date: Mon, 8 Jun 2026 00:12:45 +0700 Subject: [PATCH 23/27] fix: use Responses API with reasoning effort --- src/integrate_notes.py | 38 ++++++--------- src/spec_config.py | 1 + src/spec_llm.py | 60 +++++++----------------- tests/test_openrouter_client_timeout.py | 11 ++--- tests/test_spec_llm.py | 62 +++++++++++-------------- 5 files changed, 61 insertions(+), 111 deletions(-) diff --git a/src/integrate_notes.py b/src/integrate_notes.py index b719b6c..97adaef 100644 --- a/src/integrate_notes.py +++ b/src/integrate_notes.py @@ -29,6 +29,7 @@ ENV_API_KEY = "OPENROUTER_API_KEY" OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1" DEFAULT_MODEL = "openai/gpt-5.4" +DEFAULT_REASONING = {"effort": "medium"} DEFAULT_MAX_RETRIES = 3 RETRY_INITIAL_DELAY_SECONDS = 2.0 RETRY_BACKOFF_FACTOR = 2.0 @@ -396,25 +397,6 @@ def create_openrouter_client() -> OpenAI: ) -def _message_text(message) -> str: - content = message.content - if isinstance(content, str): - return content - if content is None: - return "" - if isinstance(content, list): - parts: List[str] = [] - for item in content: - if isinstance(item, dict): - text = item.get("text") - else: - text = getattr(item, "text", None) - if isinstance(text, str): - parts.append(text) - return "".join(parts) - return str(content) - - NOTIFY_SEND_PATH = shutil.which("notify-send") _NOTIFY_SEND_UNAVAILABLE_WARNING_EMITTED = False @@ -555,12 +537,15 @@ def build_integration_prompt( def request_integration(client: OpenAI, prompt: str, context_label: str) -> str: def perform_request() -> str: - response = client.chat.completions.create( + response = client.responses.create( model=DEFAULT_MODEL, - messages=[{"role": "user", "content": prompt}], + reasoning=DEFAULT_REASONING, + input=prompt, timeout=OPENROUTER_REQUEST_TIMEOUT_SECONDS, ) - output_text = _message_text(response.choices[0].message) + if getattr(response, "error", None): + raise RuntimeError(f"OpenRouter error for {context_label}: {response.error}") + output_text = response.output_text if not output_text.strip(): raise RuntimeError("Received empty response from GPT integration call.") patch_text = extract_patch_text_from_response(output_text) @@ -1321,12 +1306,15 @@ def build_verification_prompt( def request_verification(client: OpenAI, prompt: str, context_label: str) -> str: def perform_request() -> str: - response = client.chat.completions.create( + response = client.responses.create( model=DEFAULT_MODEL, - messages=[{"role": "user", "content": prompt}], + reasoning=DEFAULT_REASONING, + input=prompt, timeout=OPENROUTER_REQUEST_TIMEOUT_SECONDS, ) - output_text = _message_text(response.choices[0].message) + if getattr(response, "error", None): + raise RuntimeError(f"OpenRouter error for {context_label}: {response.error}") + output_text = response.output_text if not output_text.strip(): raise RuntimeError("Received empty response from GPT verification call.") return output_text.strip() diff --git a/src/spec_config.py b/src/spec_config.py index 79d004d..ad6b49a 100644 --- a/src/spec_config.py +++ b/src/spec_config.py @@ -10,6 +10,7 @@ OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1" DEFAULT_MODEL = "openai/gpt-5.4" +DEFAULT_REASONING = {"effort": "medium"} DEFAULT_MAX_RETRIES = 3 RETRY_INITIAL_DELAY_SECONDS = 2.0 diff --git a/src/spec_llm.py b/src/spec_llm.py index c372e3b..2402ca7 100644 --- a/src/spec_llm.py +++ b/src/spec_llm.py @@ -4,7 +4,7 @@ import os from dataclasses import dataclass from time import sleep -from typing import Any, Iterable +from typing import Iterable from dotenv import load_dotenv from loguru import logger @@ -13,6 +13,7 @@ from spec_config import ( DEFAULT_MAX_RETRIES, DEFAULT_MODEL, + DEFAULT_REASONING, ENV_API_KEY, OPENROUTER_BASE_URL, OPENROUTER_REQUEST_TIMEOUT_SECONDS, @@ -44,39 +45,6 @@ def create_openrouter_client() -> OpenAI: ) -def _as_chat_tool(tool: dict[str, Any]) -> dict[str, Any]: - if tool.get("type") != "function": - raise ValueError("Only function tools are supported.") - return { - "type": "function", - "function": { - "name": tool["name"], - "description": tool["description"], - "parameters": tool["parameters"], - "strict": tool.get("strict", False), - }, - } - - -def _message_text(message) -> str: - content = message.content - if isinstance(content, str): - return content - if content is None: - return "" - if isinstance(content, list): - parts = [] - for item in content: - if isinstance(item, dict): - text = item.get("text") - else: - text = getattr(item, "text", None) - if isinstance(text, str): - parts.append(text) - return "".join(parts) - return str(content) - - def execute_with_retry( operation, description: str, @@ -105,12 +73,15 @@ def execute_with_retry( def request_text(client: OpenAI, prompt: str, context_label: str) -> str: def perform_request() -> str: - response = client.chat.completions.create( + response = client.responses.create( model=DEFAULT_MODEL, - messages=[{"role": "user", "content": prompt}], + reasoning=DEFAULT_REASONING, + input=prompt, timeout=OPENROUTER_REQUEST_TIMEOUT_SECONDS, ) - output_text = _message_text(response.choices[0].message) + if getattr(response, "error", None): + raise RuntimeError(f"OpenRouter error for {context_label}: {response.error}") + output_text = response.output_text if not output_text.strip(): raise RuntimeError(f"Received empty response for {context_label}.") return output_text.strip() @@ -122,23 +93,26 @@ def request_tool_call( client: OpenAI, prompt: str, tools: Iterable[dict], context_label: str ) -> ToolCall: def perform_request() -> ToolCall: - response = client.chat.completions.create( + response = client.responses.create( model=DEFAULT_MODEL, - messages=[{"role": "user", "content": prompt}], - tools=[_as_chat_tool(tool) for tool in tools], + reasoning=DEFAULT_REASONING, + input=prompt, + tools=list(tools), tool_choice="required", parallel_tool_calls=False, timeout=OPENROUTER_REQUEST_TIMEOUT_SECONDS, ) - tool_calls = response.choices[0].message.tool_calls or [] + if getattr(response, "error", None): + raise RuntimeError(f"OpenRouter error for {context_label}: {response.error}") + tool_calls = [item for item in response.output if item.type == "function_call"] if not tool_calls: raise RuntimeError(f"No tool call returned for {context_label}.") if len(tool_calls) > 1: raise RuntimeError( f"Expected a single tool call for {context_label}, got {len(tool_calls)}." ) - function = tool_calls[0].function - return ToolCall(function.name, function.arguments) + tool_call = tool_calls[0] + return ToolCall(tool_call.name, tool_call.arguments) return execute_with_retry(perform_request, context_label) diff --git a/tests/test_openrouter_client_timeout.py b/tests/test_openrouter_client_timeout.py index 8ba6612..b42fb39 100644 --- a/tests/test_openrouter_client_timeout.py +++ b/tests/test_openrouter_client_timeout.py @@ -62,18 +62,15 @@ def test_integration_request_sets_per_call_timeout() -> None: f"{integrate_notes.PATCH_BLOCK_END}" ) - class Completions: + class Responses: def create(self, **kwargs): captured_kwargs.update(kwargs) - return SimpleNamespace( - choices=[ - SimpleNamespace(message=SimpleNamespace(content=patch_response)) - ] - ) + return SimpleNamespace(error=None, output_text=patch_response) - client = SimpleNamespace(chat=SimpleNamespace(completions=Completions())) + client = SimpleNamespace(responses=Responses()) assert integrate_notes.request_integration(client, "prompt", "unit") + assert captured_kwargs["reasoning"] == integrate_notes.DEFAULT_REASONING assert ( captured_kwargs["timeout"] == integrate_notes.OPENROUTER_REQUEST_TIMEOUT_SECONDS diff --git a/tests/test_spec_llm.py b/tests/test_spec_llm.py index 2d94f66..fe03dd8 100644 --- a/tests/test_spec_llm.py +++ b/tests/test_spec_llm.py @@ -8,53 +8,52 @@ import spec_llm # noqa: E402 from spec_config import DEFAULT_MODEL # noqa: E402 +from spec_config import DEFAULT_REASONING # noqa: E402 from spec_config import OPENROUTER_REQUEST_TIMEOUT_SECONDS # noqa: E402 -class FakeCompletions: - def __init__(self, message): - self.message = message +class FakeResponses: + def __init__(self, response): + self.response = response self.kwargs = None def create(self, **kwargs): self.kwargs = kwargs - return SimpleNamespace(choices=[SimpleNamespace(message=self.message)]) + return self.response -def fake_client(message): - completions = FakeCompletions(message) - client = SimpleNamespace( - chat=SimpleNamespace(completions=completions), - ) - return client, completions +def fake_client(response): + responses = FakeResponses(response) + client = SimpleNamespace(responses=responses) + return client, responses -def test_request_text_uses_openrouter_chat_completion_shape(): - client, completions = fake_client(SimpleNamespace(content=" done ")) +def test_request_text_uses_openrouter_responses_shape(): + client, responses = fake_client(SimpleNamespace(error=None, output_text=" done ")) result = spec_llm.request_text(client, "prompt", "unit") assert result == "done" - assert completions.kwargs == { + assert responses.kwargs == { "model": DEFAULT_MODEL, - "messages": [{"role": "user", "content": "prompt"}], + "reasoning": DEFAULT_REASONING, + "input": "prompt", "timeout": OPENROUTER_REQUEST_TIMEOUT_SECONDS, } -def test_request_tool_call_converts_response_tool_schema_to_chat_tool_schema(): - message = SimpleNamespace( - content=None, - tool_calls=[ +def test_request_tool_call_uses_openrouter_responses_tool_shape(): + response = SimpleNamespace( + error=None, + output=[ SimpleNamespace( - function=SimpleNamespace( - name="edit_notes", - arguments='{"action":"edit","edits":[]}', - ) + type="function_call", + name="edit_notes", + arguments='{"action":"edit","edits":[]}', ) ], ) - client, completions = fake_client(message) + client, responses = fake_client(response) tool_schema = { "type": "function", "name": "edit_notes", @@ -67,20 +66,11 @@ def test_request_tool_call_converts_response_tool_schema_to_chat_tool_schema(): assert tool_call.name == "edit_notes" assert tool_call.arguments == '{"action":"edit","edits":[]}' - assert completions.kwargs == { + assert responses.kwargs == { "model": DEFAULT_MODEL, - "messages": [{"role": "user", "content": "prompt"}], - "tools": [ - { - "type": "function", - "function": { - "name": "edit_notes", - "description": "Edit checked-out notes.", - "parameters": {"type": "object", "properties": {}}, - "strict": True, - }, - } - ], + "reasoning": DEFAULT_REASONING, + "input": "prompt", + "tools": [tool_schema], "tool_choice": "required", "parallel_tool_calls": False, "timeout": OPENROUTER_REQUEST_TIMEOUT_SECONDS, From fed1bbad918d710fc8d0f4ae589bbb616e65f878 Mon Sep 17 00:00:00 2001 From: distbit Date: Mon, 8 Jun 2026 00:16:26 +0700 Subject: [PATCH 24/27] fix: use high reasoning effort --- src/integrate_notes.py | 2 +- src/spec_config.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/integrate_notes.py b/src/integrate_notes.py index 97adaef..88a8be0 100644 --- a/src/integrate_notes.py +++ b/src/integrate_notes.py @@ -29,7 +29,7 @@ ENV_API_KEY = "OPENROUTER_API_KEY" OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1" DEFAULT_MODEL = "openai/gpt-5.4" -DEFAULT_REASONING = {"effort": "medium"} +DEFAULT_REASONING = {"effort": "high"} DEFAULT_MAX_RETRIES = 3 RETRY_INITIAL_DELAY_SECONDS = 2.0 RETRY_BACKOFF_FACTOR = 2.0 diff --git a/src/spec_config.py b/src/spec_config.py index ad6b49a..b170978 100644 --- a/src/spec_config.py +++ b/src/spec_config.py @@ -10,7 +10,7 @@ OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1" DEFAULT_MODEL = "openai/gpt-5.4" -DEFAULT_REASONING = {"effort": "medium"} +DEFAULT_REASONING = {"effort": "high"} DEFAULT_MAX_RETRIES = 3 RETRY_INITIAL_DELAY_SECONDS = 2.0 From d58fb6d70c787bddc9e848f30cfb3d24a8e5c463 Mon Sep 17 00:00:00 2001 From: distbit Date: Mon, 8 Jun 2026 00:49:12 +0700 Subject: [PATCH 25/27] integrate_notes.py test_openrouter_client_timeout.py --- .codex | 0 src/integrate_notes.py | 283 +++++++++++++----------- tests/test_openrouter_client_timeout.py | 46 +++- 3 files changed, 189 insertions(+), 140 deletions(-) create mode 100644 .codex diff --git a/.codex b/.codex new file mode 100644 index 0000000..e69de29 diff --git a/src/integrate_notes.py b/src/integrate_notes.py index 88a8be0..b4f62ff 100644 --- a/src/integrate_notes.py +++ b/src/integrate_notes.py @@ -28,7 +28,7 @@ DEFAULT_CHUNK_MAX_WORDS = 400 ENV_API_KEY = "OPENROUTER_API_KEY" OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1" -DEFAULT_MODEL = "openai/gpt-5.4" +DEFAULT_MODEL = "minimax/minimax-m3" DEFAULT_REASONING = {"effort": "high"} DEFAULT_MAX_RETRIES = 3 RETRY_INITIAL_DELAY_SECONDS = 2.0 @@ -93,6 +93,49 @@ PATCH_BLOCK_END = ">>>>>>> REPLACE" DUPLICATION_BLOCK_START = "<<<<<<< DUPLICATE" DUPLICATION_BLOCK_END = ">>>>>>> DUPLICATE" +INTEGRATION_RESPONSE_SCHEMA = { + "type": "object", + "properties": { + "action": {"type": "string", "enum": ["integrate"]}, + "patches": { + "type": "array", + "items": { + "type": "object", + "properties": { + "search": {"type": "string"}, + "replace": {"type": "string"}, + }, + "required": ["search", "replace"], + "additionalProperties": False, + }, + }, + "duplications": { + "type": "array", + "items": { + "type": "object", + "properties": { + "notes": {"type": "string"}, + "body": {"type": "string"}, + }, + "required": ["notes", "body"], + "additionalProperties": False, + }, + }, + }, + "required": ["action", "patches", "duplications"], + "additionalProperties": False, +} +INTEGRATION_RESPONSE_FORMAT = { + "type": "json_schema", + "name": "integration_response", + "strict": True, + "schema": INTEGRATION_RESPONSE_SCHEMA, +} +INTEGRATION_RESPONSE_SCHEMA_TEXT = json.dumps( + INTEGRATION_RESPONSE_SCHEMA, + ensure_ascii=False, + indent=2, +) MAX_PATCH_ATTEMPTS = 3 @@ -470,16 +513,17 @@ def build_integration_prompt( f"Maintain the grouping approach: {grouping}." ) response_instructions = ( - "Return only patch instructions and duplication proofs using the exact structures below." - " For each patch:" - f"\n{PATCH_BLOCK_START}\n\n{PATCH_BLOCK_DIVIDER}\n\n{PATCH_BLOCK_END}" - "\nFor each duplication proof (use this when notes are already present in the current document body, so no patch is needed):" - f"\n{DUPLICATION_BLOCK_START}\n\n{PATCH_BLOCK_DIVIDER}\n\n{DUPLICATION_BLOCK_END}" - "\nEmit the blocks back-to-back in the order they should be applied or checked. " - "If any notes are already present in the document body and therefore do not need a patch, you must include a duplication proof block for them. " + "Return exactly one JSON object matching the schema below. " + "Use patches for edits and duplications for notes already present in the current document body. " + "For each patch, search must be the exact text to find and replace must be the complete replacement text. " + "For insertions, include the anchor text in both search and replace. " + "For each duplication proof, notes must be the exact notes text already covered and body must be the exact current document body text that contains it. " + "Emit patches in the order they should be applied. " + "If any notes are already present in the document body and therefore do not need a patch, you must include a duplication proof entry for them. " "SEARCH and DUPLICATE/BODY text must each be a single contiguous span copied from the current document body; do not concatenate separate sections. " - "Do not add commentary, numbering, markdown fences, or explanations. " - "If no changes are required and no duplication proofs are needed, return an empty string." + "Do not add commentary, numbering, markdown fences, or explanations outside the JSON object. " + "Use empty patches and duplications arrays only when no changes are required and no duplication proofs are needed." + f"\n\n{INTEGRATION_RESPONSE_SCHEMA_TEXT}\n" ) sections = [ @@ -496,12 +540,12 @@ def build_integration_prompt( feedback_lines: List[str] = [] if failed_formatting: feedback_lines.append( - "The previous response could not be parsed. Fix the formatting issues below and re-emit only valid blocks." + "The previous JSON response could not be parsed. Fix the issues below and re-emit only a valid JSON object." ) feedback_lines.append(f"Error: {failed_formatting}") if failed_patches: feedback_lines.append( - "The previous patch attempt failed because the SEARCH block(s) below did not match the current document." + "The previous patch attempt failed because the search text below did not match the current document." ) for failure in failed_patches: feedback_lines.append( @@ -527,9 +571,9 @@ def build_integration_prompt( if previous_response: sections.append( - "\n" + "\n" + previous_response - + "\n" + + "\n" ) return "\n\n\n\n\n".join(sections) @@ -541,6 +585,7 @@ def perform_request() -> str: model=DEFAULT_MODEL, reasoning=DEFAULT_REASONING, input=prompt, + text={"format": INTEGRATION_RESPONSE_FORMAT}, timeout=OPENROUTER_REQUEST_TIMEOUT_SECONDS, ) if getattr(response, "error", None): @@ -548,30 +593,11 @@ def perform_request() -> str: output_text = response.output_text if not output_text.strip(): raise RuntimeError("Received empty response from GPT integration call.") - patch_text = extract_patch_text_from_response(output_text) - # logger.debug(f"Integration patches for {context_label}:\n{patch_text}") - return patch_text + return output_text.strip() return execute_with_retry(perform_request, f"integration {context_label}") -def extract_patch_text_from_response(response_text: str) -> str: - stripped = response_text.strip() - if not stripped: - return "" - - if stripped.startswith("```"): - lines = stripped.splitlines() - if not lines: - return "" - lines = lines[1:] - while lines and lines[-1].strip() == "```": - lines.pop() - stripped = "\n".join(lines).strip() - - return stripped - - @dataclass(frozen=True) class PatchInstruction: search_text: str @@ -604,123 +630,116 @@ def __init__(self, message: str, block_text: str) -> None: self.block_text = block_text -def _normalize_line_endings(text: str) -> str: - return text.replace("\r\n", "\n").replace("\r", "\n") - +def _json_payload_text(payload: Any) -> str: + try: + return json.dumps(payload, ensure_ascii=False, indent=2) + except TypeError: + return repr(payload) -def _sanitize_patch_segment(segment: str) -> str: - cleaned = _normalize_line_endings(segment) - if cleaned.startswith("\n"): - cleaned = cleaned[1:] - if cleaned.endswith("\n"): - cleaned = cleaned[:-1] - return cleaned +def _strip_json_code_fence(response_text: str) -> str: + stripped = response_text.strip() + if not stripped.startswith("```"): + return stripped -def _find_next_block_start( - text: str, position: int, markers: Sequence[str] -) -> tuple[int, str] | None: - candidates: List[tuple[int, str]] = [] - for marker in markers: - index = text.find(marker, position) - if index != -1: - candidates.append((index, marker)) - if not candidates: - return None - return min(candidates, key=lambda item: item[0]) + lines = stripped.splitlines() + if len(lines) < 3: + return stripped + opening = lines[0].strip().lower() + if opening not in {"```", "```json"} or lines[-1].strip() != "```": + return stripped -def _slice_block_for_error(text: str, start_index: int, search_from: int) -> str: - next_block = _find_next_block_start( - text, search_from, [PATCH_BLOCK_START, DUPLICATION_BLOCK_START] + logger.warning( + "Integration response was wrapped in a markdown JSON fence; parsing the fenced JSON body." ) - end_index = next_block[0] if next_block else len(text) - return text[start_index:end_index].strip() + return "\n".join(lines[1:-1]).strip() -def parse_integration_blocks( +def parse_integration_payload( response_text: str, ) -> tuple[List[PatchInstruction], List[DuplicationProof]]: if not response_text.strip(): - return [], [] - - cleaned = _normalize_line_endings(response_text) - instructions: List[PatchInstruction] = [] - duplications: List[DuplicationProof] = [] - position = 0 + raise IntegrationParseError("Integration response is empty.", response_text) + json_text = _strip_json_code_fence(response_text) + try: + payload = json.loads(json_text) + except json.JSONDecodeError as error: + raise IntegrationParseError( + f"Integration response is not valid JSON: {error}", + response_text, + ) from error + if not isinstance(payload, dict): + raise IntegrationParseError( + "Integration response must be a JSON object.", + _json_payload_text(payload), + ) - while True: - next_block = _find_next_block_start( - cleaned, position, [PATCH_BLOCK_START, DUPLICATION_BLOCK_START] + if payload.get("action") != "integrate": + raise IntegrationParseError( + "Integration JSON response must include action='integrate'.", + _json_payload_text(payload), ) - if next_block is None: - remaining = cleaned[position:].strip() - if remaining: - logger.warning( - "Ignoring unexpected content outside integration blocks: {}".format( - remaining[:120] - ) - ) - break - start_index, block_start = next_block - divider_index = cleaned.find( - PATCH_BLOCK_DIVIDER, start_index + len(block_start) + patches = payload.get("patches") + if not isinstance(patches, list): + raise IntegrationParseError( + "Integration JSON response must include a patches array.", + _json_payload_text(payload), + ) + duplication_payloads = payload.get("duplications") + if not isinstance(duplication_payloads, list): + raise IntegrationParseError( + "Integration JSON response must include a duplications array.", + _json_payload_text(payload), ) - if divider_index == -1: - block_text = _slice_block_for_error( - cleaned, start_index, start_index + len(block_start) - ) - raise IntegrationParseError( - "Integration block is missing the divider '{}'.".format( - PATCH_BLOCK_DIVIDER - ), - block_text, - ) - if block_start == PATCH_BLOCK_START: - block_end = PATCH_BLOCK_END - else: - block_end = DUPLICATION_BLOCK_END + instructions: List[PatchInstruction] = [] + duplications: List[DuplicationProof] = [] - end_index = cleaned.find(block_end, divider_index + len(PATCH_BLOCK_DIVIDER)) - if end_index == -1: - block_text = _slice_block_for_error( - cleaned, start_index, divider_index + len(PATCH_BLOCK_DIVIDER) + for patch in patches: + if not isinstance(patch, dict): + raise IntegrationParseError( + "Each patch must be an object.", + _json_payload_text(payload), ) + search_text = patch.get("search") + if not isinstance(search_text, str) or not search_text.strip(): raise IntegrationParseError( - "Integration block is missing the end marker '{}'.".format(block_end), - block_text, + "Each patch must include non-empty search text.", + _json_payload_text(payload), ) + replace_text = patch.get("replace") + if not isinstance(replace_text, str): + raise IntegrationParseError( + "Each patch must include string replace text.", + _json_payload_text(payload), + ) + instructions.append( + PatchInstruction(search_text=search_text, replace_text=replace_text) + ) - first_segment = cleaned[start_index + len(block_start) : divider_index] - second_segment = cleaned[divider_index + len(PATCH_BLOCK_DIVIDER) : end_index] - - first_text = _sanitize_patch_segment(first_segment) - second_text = _sanitize_patch_segment(second_segment) - - if block_start == PATCH_BLOCK_START: - if not first_text.strip(): - block_text = cleaned[start_index : end_index + len(block_end)].strip() - raise IntegrationParseError( - "Patch SEARCH text must contain non-whitespace characters.", - block_text, - ) - instructions.append( - PatchInstruction(search_text=first_text, replace_text=second_text) + for duplication_payload in duplication_payloads: + if not isinstance(duplication_payload, dict): + raise IntegrationParseError( + "Each duplication proof must be an object.", + _json_payload_text(payload), ) - else: - if not first_text.strip() or not second_text.strip(): - block_text = cleaned[start_index : end_index + len(block_end)].strip() - raise IntegrationParseError( - "Duplication block must include non-whitespace notes and body text.", - block_text, - ) - duplications.append( - DuplicationProof(notes_text=first_text, body_text=second_text) + notes_text = duplication_payload.get("notes") + body_text = duplication_payload.get("body") + if not isinstance(notes_text, str) or not notes_text.strip(): + raise IntegrationParseError( + "Each duplication proof must include non-empty notes text.", + _json_payload_text(payload), ) - - position = end_index + len(block_end) + if not isinstance(body_text, str) or not body_text.strip(): + raise IntegrationParseError( + "Each duplication proof must include non-empty body text.", + _json_payload_text(payload), + ) + duplications.append( + DuplicationProof(notes_text=notes_text, body_text=body_text) + ) return instructions, duplications @@ -946,11 +965,11 @@ def integrate_chunk_with_patches( else None ), ) - patch_text = request_integration(client, prompt, attempt_label) - previous_response = patch_text + response_text = request_integration(client, prompt, attempt_label) + previous_response = response_text try: - instructions, duplications = parse_integration_blocks(patch_text) + instructions, duplications = parse_integration_payload(response_text) except IntegrationParseError as error: failed_formatting = str(error) failed_patches = None @@ -960,7 +979,7 @@ def integrate_chunk_with_patches( ) logger.info( f"Invalid integration response for {attempt_label} on attempt {attempt}; " - f"reason: {error}\nFailed block:\n{error.block_text}" + f"reason: {error}\nFailed payload:\n{error.block_text}" ) logger.info( f"Retrying {context_label}; response formatting was invalid on attempt {attempt}." diff --git a/tests/test_openrouter_client_timeout.py b/tests/test_openrouter_client_timeout.py index b42fb39..d0d2dfb 100644 --- a/tests/test_openrouter_client_timeout.py +++ b/tests/test_openrouter_client_timeout.py @@ -1,3 +1,4 @@ +import json import sys from pathlib import Path from types import SimpleNamespace @@ -54,23 +55,52 @@ def openai_client(**kwargs): def test_integration_request_sets_per_call_timeout() -> None: captured_kwargs = {} - patch_response = ( - f"{integrate_notes.PATCH_BLOCK_START}\n" - "# Body\n" - f"{integrate_notes.PATCH_BLOCK_DIVIDER}\n" - "# Body\n" - f"{integrate_notes.PATCH_BLOCK_END}" + tool_arguments = json.dumps( + { + "action": "integrate", + "patches": [ + { + "search": "- Create space for the other person to talk.\n- Ask open-ended follow-up questions.", + "replace": ( + "- Create space for the other person to talk.\n" + "- Ask people to tell you more rather than immediately giving advice or your opinion.\n" + '- Use verbal acknowledgments while they are speaking, e.g., "yeah that makes sense," "uh huh."\n' + "- Ask open-ended follow-up questions." + ), + } + ], + "duplications": [], + } ) class Responses: def create(self, **kwargs): captured_kwargs.update(kwargs) - return SimpleNamespace(error=None, output_text=patch_response) + return SimpleNamespace(error=None, output_text=f"```json\n{tool_arguments}\n```") client = SimpleNamespace(responses=Responses()) - assert integrate_notes.request_integration(client, "prompt", "unit") + response_text = integrate_notes.request_integration(client, "prompt", "unit") + instructions, duplications = integrate_notes.parse_integration_payload( + response_text + ) + + assert instructions == [ + integrate_notes.PatchInstruction( + search_text="- Create space for the other person to talk.\n- Ask open-ended follow-up questions.", + replace_text=( + "- Create space for the other person to talk.\n" + "- Ask people to tell you more rather than immediately giving advice or your opinion.\n" + '- Use verbal acknowledgments while they are speaking, e.g., "yeah that makes sense," "uh huh."\n' + "- Ask open-ended follow-up questions." + ), + ) + ] + assert duplications == [] assert captured_kwargs["reasoning"] == integrate_notes.DEFAULT_REASONING + assert captured_kwargs["text"] == { + "format": integrate_notes.INTEGRATION_RESPONSE_FORMAT + } assert ( captured_kwargs["timeout"] == integrate_notes.OPENROUTER_REQUEST_TIMEOUT_SECONDS From 3d031fd0896a841acf8543baf79ad74ed67468c2 Mon Sep 17 00:00:00 2001 From: distbit Date: Mon, 8 Jun 2026 01:00:59 +0700 Subject: [PATCH 26/27] chore: ignore local codex marker --- .codex | 0 .gitignore | 1 + 2 files changed, 1 insertion(+) delete mode 100644 .codex diff --git a/.codex b/.codex deleted file mode 100644 index e69de29..0000000 diff --git a/.gitignore b/.gitignore index bf10399..e42ba3a 100644 --- a/.gitignore +++ b/.gitignore @@ -16,6 +16,7 @@ pip-wheel-metadata/ share/python-wheels/ MANIFEST .aider* +.codex # Log files *.log From 4a8159178a034844333354f493064fb27d9ddec2 Mon Sep 17 00:00:00 2001 From: distbit Date: Mon, 8 Jun 2026 01:01:58 +0700 Subject: [PATCH 27/27] docs: record M3 JSON integration behavior --- CONTEXT_LOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CONTEXT_LOG.md b/CONTEXT_LOG.md index 449bbd8..f77405a 100644 --- a/CONTEXT_LOG.md +++ b/CONTEXT_LOG.md @@ -15,4 +15,5 @@ - `continuous-note-organisation.timer` runs `src/integrate_notes.py --continuous --notes-root /home/pimania/notes` daily at 09:00 as a systemd user timer. The timer unit is stored outside this repo under `~/.config/systemd/user/`. - The timer depends on Git hooks in `/home/pimania/notes` being able to find `git-lfs`. On 2026-06-07, continuous integration failed at the pre-integration `git push` because user systemd's PATH omitted Homebrew (`/home/linuxbrew/.linuxbrew/bin`), where `git-lfs` is installed. The persistent fix is `~/.config/environment.d/10-user-path.conf`; the running user manager was also updated with `systemctl --user set-environment`. - On 2026-06-07, manual reruns with an uncommitted `DEFAULT_MODEL = "minimax/minimax-m3"` change reached the LLM step but failed chunk 1 because the model repeatedly emitted malformed patch blocks, including extra `SEARCH` text inside the search span and a combined `<<<<<<< SEARCH DUPLICATE` marker. The script correctly refused to write these patches. +- On 2026-06-08, `minimax/minimax-m3` with high reasoning effort still produced unreliable free-form patch formatting. OpenRouter returned 404 for M3 Responses calls using required tool calls, and accepted but did not enforce `text.format` JSON schema by itself, so continuous integration now prompts for strict JSON over Responses and validates it locally. M3 commonly wraps valid JSON in a top-level markdown JSON fence despite instructions; the parser accepts only that wrapper with a warning, then still validates the JSON fields strictly. - Continuous mode now writes a default `grouping: |` frontmatter value, with a warning log, when a note is marked `organise: continuous` but has no grouping. This keeps scheduled runs non-interactive while making the default explicit in the note.