Add spec-based integration flow

.gitignore

@@ -16,6 +16,7 @@ pip-wheel-metadata/
 share/python-wheels/
 MANIFEST
 .aider*
+.codex
 
 # Log files
 *.log
@@ -94,4 +95,5 @@ src/pending_verification_prompts.json
 
 
 src/logs/*
-logs/*
+logs/*
+**/node_modules/

.python-version

@@ -0,0 +1 @@
+3.13

.vscode/settings.json

@@ -1,6 +1,6 @@
 {
-  "python.terminal.activateEnvironment": true,
-  "python.terminal.activateEnvInCurrentTerminal": true,
   "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
-  "python.analysis.extraPaths": ["${workspaceFolder}/src"]
+  "python.analysis.extraPaths": [
+    "${workspaceFolder}/src"
+  ]
 }

CONTEXT_LOG.md

@@ -0,0 +1,19 @@
+# Runtime Compatibility
+
+- The dependency lock currently resolves `pydantic-core==2.33.2`, which ships Python 3.13 wheels but falls back to a PyO3 source build on Python 3.14. PyO3 0.24.1 rejects 3.14, so the project needs to stay pinned to Python 3.13 until the dependency set is upgraded to a 3.14-compatible release line.
+
+# OpenRouter Env Routing
+
+- OpenRouter clients load the repo `.env` with override enabled so the repo-specific `OPENROUTER_API_KEY` wins over inherited shell values. This prevents global keys from merging Integrate Notes usage with other projects.
+## Notes filename slug migration
+
+- The notes vault uses hyphen-slug filenames while keeping readable wikilinks. Note resolution and exploration file selection should compare by slug-normalized note title rather than exact filename text.
+
+## Note directive frontmatter
+
+- The grouping directive is now frontmatter (`grouping: |`) rather than body text near the top of notes. Continuous scratchpad integration is opt-in via `organise: continuous`; pending continuous notes must also have `grouping` frontmatter so batch mode does not prompt mid-run.
+- `continuous-note-organisation.timer` runs `src/integrate_notes.py --continuous --notes-root /home/pimania/notes` daily at 09:00 as a systemd user timer. The timer unit is stored outside this repo under `~/.config/systemd/user/`.
+- The timer depends on Git hooks in `/home/pimania/notes` being able to find `git-lfs`. On 2026-06-07, continuous integration failed at the pre-integration `git push` because user systemd's PATH omitted Homebrew (`/home/linuxbrew/.linuxbrew/bin`), where `git-lfs` is installed. The persistent fix is `~/.config/environment.d/10-user-path.conf`; the running user manager was also updated with `systemctl --user set-environment`.
+- On 2026-06-07, manual reruns with an uncommitted `DEFAULT_MODEL = "minimax/minimax-m3"` change reached the LLM step but failed chunk 1 because the model repeatedly emitted malformed patch blocks, including extra `SEARCH` text inside the search span and a combined `<<<<<<< SEARCH DUPLICATE` marker. The script correctly refused to write these patches.
+- On 2026-06-08, `minimax/minimax-m3` with high reasoning effort still produced unreliable free-form patch formatting. OpenRouter returned 404 for M3 Responses calls using required tool calls, and accepted but did not enforce `text.format` JSON schema by itself, so continuous integration now prompts for strict JSON over Responses and validates it locally. M3 commonly wraps valid JSON in a top-level markdown JSON fence despite instructions; the parser accepts only that wrapper with a warning, then still validates the JSON fields strictly.
+- Continuous mode now writes a default `grouping: |` frontmatter value, with a warning log, when a note is marked `organise: continuous` but has no grouping. This keeps scheduled runs non-interactive while making the default explicit in the note.

INTERVIEW.md

@@ -1,26 +0,0 @@
-# TODO 7: Multiline grouping approach (excluded from LLM body, protected from patches)
-
-> What exact on-disk syntax do you want for the multiline grouping section? Please provide a concrete before/after example (including where it sits relative to `---` front matter and the `# -- SCRATCHPAD` heading).
-
-after the front matter, before the scratchpad heading. it should be before any other content in the file after the front matter
-
-> How should the end of the multiline grouping section be detected (e.g., first blank line, next heading, a closing marker, end-of-file)? Can the grouping text itself contain blank lines?
-
-use explicit opening and closing marker syntax. if markdown frontmatter satisfies the criteria e.g. support for multiline content, use that. otherwise implement this in the way which makes the most sense / is the most aligned w/ good practice.
-
-> For legacy documents that still use a single-line `Grouping approach: ...` prefix, should the tool leave that line as-is, or migrate it to the new multiline format when writing the file? If migration is desired, should it happen only when `--grouping` is provided / user is prompted, or always?
-
-leave as is 
-
-> The current CLI prompt uses `input()` (single line). How do you want multiline grouping input to be entered (e.g., read until a lone `.` line, read until EOF, open $EDITOR, allow literal `\n` escapes, etc.)?
-
-yes figure out best/most simple but also useable way to support multiline input
-
-> Should the grouping section be preserved verbatim (whitespace/indentation), or normalized (trim lines, collapse spaces) before inserting into the prompt’s “Maintain the grouping approach: …” line?
-
-preserved verbatim 
-
-> Do you want the grouping section to be strictly immutable during patch application (i.e., patches only apply to the body after removing the grouping block), or should we also detect and error if a patch’s SEARCH text matches inside the grouping section?
-
-strictly immutable. it should be as if it didn't exist in the body at all. so if a search block matches it and nothing else, it results in an error > retry. there should not need to be any special handling logic for these cases. it simply isn't part of the document body for the purposes of search/replace or substitute blocks.
-

TODO.md

@@ -1,7 +1,13 @@
 - ask model to provide snippets from integrate text for each find/replace, to clarify what it intended it to integrate
     - on fail, only ask the model to provide just that single failed block instead of asking it to provide all blocks again. this is only possible once the model returns what integrated text each block relates 
 - only ask for start and end lines of search block instead of exact text. if there are multiple matches, ask model to provide a sufficient number of of lines at start or end to narrow down options to a single match. ensure that the search block which the verification prompt sees is not affected by this, by populating the SEARCH section of the block given in the verification prompt with the matching text from the file, instead of only including the start and end lines provided by the model
-- modify so that it uses tool calling + hierarchical markdown parsing to avoid needing to ever send the entire document to the model, and instead allow the model to find the relevant section(s) (could often be more than one section which should be modified even to integrate a single piece of info) to modify, and then once it has found the sections, it provides the search/replace diffs
-    - ensure that the model uses arbitrarily nested md headers, to make this approach scalable instead of only e.g. using one level of headings
-- move logs outside of src/
-- put group strat into front matter
+
+
+- move all files in this repo except integrate_notes.py to a new repo next to this one called zettel_inbox. still make sure to leave a copy of all non-code files in this repo, while also carrying them across.
+- add a new tool, in addition to find/replace, for creating a new file. this just takes a file name to create and initial text to add. add to the instructions an explanation that if the model wants to put some of the content from the chunk into a new file and link to it, because it couldn't find anywhere else satisfactory, it should just use the file tool to create the file, and then link to the file using a wikilink [[file_name]]. importantly it must always link to any new files it creates in at least one place (add validation rule for this). enforce that it must write some new text to any new file.
+
+- figure out how to make my notes work with butter context repo. Add symlinks automatically for all files under infofi into a dir in the butter repo. Check that this supports two way sync
+- add guidance re maximum size of file
+- add a copy and paste after function. also log all actions
+- prevent it from adding text to index gists. Only new files
+- use gemini flash

pyproject.toml

@@ -12,7 +12,7 @@ dependencies = [
     "requests",
     "openai>=1.109.1",
 ]
-requires-python = ">=3.13"
+requires-python = ">=3.13,<3.14"
 license = {text = "UNLICENCE"}
 readme = "README.md"
 
@@ -25,3 +25,8 @@ allow-direct-references = true
 
 [tool.hatch.build.targets.wheel]
 packages = ["src"]
+
+[dependency-groups]
+dev = [
+    "pytest>=9.0.3",
+]