diff --git a/CHANGELOG.md b/CHANGELOG.md index 84d865a6a..58477b68d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ Full release notes with details on each version: [GitHub Releases](https://githu ## 0.9.17 (unreleased) +- Fix: the semantic cache no longer replays extractions from an older prompt after an upgrade (#1939, thanks @HunterMcGrew). Entries were keyed on `sha256(file content + path)` alone, with no component for the extraction prompt that produced them, so a release that changed the prompt left every unchanged file a cache hit: the run exited 0, `cost.json` looked cheap, and the graph silently carried two prompt generations side by side. Semantic entries are now namespaced by a fingerprint of the extraction prompt (`cache/semantic/p{fingerprint}/`, mirroring the AST cache's `v{version}/` layout), keeping both properties #1252 wanted — entries survive releases that don't touch the prompt, and invalidate only when it actually changed. The fingerprint normalizes line endings so a CRLF checkout doesn't look like a prompt change. Both extraction paths pass their prompt: the Python/CLI path (`llm.py`'s `_EXTRACTION_SYSTEM`, all backends) automatically, and the skill path via a new `prompt_file` argument in Step B0/B3 pointing at the `references/extraction-spec.md` the subagents were handed. Pre-existing entries predate fingerprinting and have unknowable vintage: they are still served rather than re-billing a whole corpus, but `check_semantic_cache` now warns with the count, so the "no signal at all" the report describes becomes a visible one; `--force` (or `GRAPHIFY_FORCE=1`) re-extracts them. Old-fingerprint entries are pruned by liveness only, never swept wholesale the way stale AST versions are — two hosts with different prompts can share one `graphify-out/`, and a wholesale sweep would have each run delete the other's entries. (The two monolith skills, aider and devin, inline their prompt instead of shipping a spec sidecar and stay on the unfingerprinted path for now.) - Fix: a missing `manifest.json` no longer degrades `graphify extract --code-only` into a full scan that discards the committed semantic layer (#1925). On a fresh clone (or when the manifest is deliberately untracked because its mtimes churn), the incremental gate required both `manifest.json` and `graph.json`; with only the graph present it fell to a full scan, and under `--code-only` that dropped every doc/paper/image node — silently replacing a curated graph with an AST-only skeleton. An existing `graph.json` is now a sufficient incremental baseline: `detect_incremental` already treats an absent manifest as "everything new / nothing deleted", so `build_merge` + `_stale_graph_sources` preserve files that are merely out of this run's scope while still evicting genuinely deleted sources. - Fix: hyperedge-only documents are now stamped in the manifest instead of being re-extracted on every run (#1920). `_stamped_manifest_files` (#1897) decided a semantic doc "produced output" by inspecting only `nodes` and `edges`, never `hyperedges`, so a chunk whose only output for a doc was a hyperedge (3+ nodes sharing a concept) left that doc unstamped and perpetually re-queued. Stamping now counts hyperedge output too, mirroring the per-`source_file` keying the semantic cache already uses. - Fix: the PHP extractor now disambiguates same-named classes across namespaces (#1923). A bare class reference (`extends Page`) collapsed onto the only internal class named `Page`, so `App\Models\Page` and an imported `Filament\Pages\Page` fused into one node, manufacturing a false `inherits`/`imports` edge and a bogus cross-community bridge. A new namespace/`use`-aware resolution pass (mirroring the Java resolver, running before the unique-name rewire) re-points supertype/import references to the real definition, or parks provably-external ones on a fully-qualified stub the bare-name rewire cannot collapse. Plain, non-namespaced PHP is unchanged. diff --git a/graphify/cache.py b/graphify/cache.py index 15795d441..7122b5c58 100644 --- a/graphify/cache.py +++ b/graphify/cache.py @@ -63,6 +63,99 @@ def _cleanup_stale_ast_entries(ast_base: Path, current_dir: Path) -> None: pass +# Semantic cache entries are LLM output, so they depend on the extraction prompt +# that produced them, not just on file contents. Keying purely on content means a +# release that changes the prompt keeps replaying entries from the older prompt on +# every unchanged file, silently mixing extraction vintages in one graph (#1939). +# Versioning them by package version (as the AST cache does) would re-bill LLM +# extraction on every patch release — the reason #1252 deliberately left them +# unversioned. Fingerprinting the prompt itself keeps both properties: entries +# survive releases that don't touch the prompt, and invalidate only when it +# actually changed. Entries live under cache/semantic/p{fingerprint}/ when the +# caller supplies its prompt; callers that don't keep the historical flat layout. +_PROMPT_FP_LEN = 12 + +# Count of pre-fingerprint (flat-layout) entries served this process, so +# check_semantic_cache can report N to the user (#1939). +_legacy_semantic_hits = 0 + +# Prompt-file fingerprints already computed, keyed by (path, size, mtime_ns) — +# the same stat signature the hash index uses. check_semantic_cache resolves the +# prompt once per FILE in the corpus, so without this a 500-doc run re-reads and +# re-hashes the same spec 500 times (and warns 500 times when it is unreadable). +_prompt_fp_cache: dict[tuple, str] = {} + + +def prompt_fingerprint(prompt: "str | Path") -> str: + """Return a short stable fingerprint of an extraction prompt. + + ``prompt`` is either the prompt text itself (the Python extraction path owns + its system prompt, :func:`graphify.llm._extraction_system`) or a Path to the + prompt file an agent loaded (the skill path's + ``references/extraction-spec.md``). + + Line endings and trailing whitespace are normalized before hashing: the same + spec file checked out with CRLF on Windows must not fingerprint differently + from the LF checkout that wrote the cache, or every Windows run would look + like a prompt change and re-bill extraction. + """ + if isinstance(prompt, Path): + text = prompt.read_text(encoding="utf-8", errors="replace") + else: + text = prompt + normalized = "\n".join( + line.rstrip() for line in text.replace("\r\n", "\n").replace("\r", "\n").split("\n") + ).strip() + return hashlib.sha256(normalized.encode()).hexdigest()[:_PROMPT_FP_LEN] + + +def _resolve_prompt_fp(prompt: "str | Path | None" = None, + prompt_file: "str | Path | None" = None) -> str | None: + """Fingerprint the caller's extraction prompt, or None when it supplied none. + + ``prompt`` is prompt TEXT; ``prompt_file`` is a path to a file CONTAINING the + prompt. They are separate parameters rather than one overloaded argument + because the skill-driven callers are markdown snippets an agent copies with a + path substituted in — passing that path as ``prompt`` would hash the path + string itself, yielding a fingerprint that is stable, plausible, and tracks + nothing about the prompt. A silent wrong fingerprint is the exact failure + class #1939 is about, so the two are not inferred from each other. + + Best-effort: an unreadable ``prompt_file`` falls back to the flat, unattributed + layout rather than failing the run — a cache is never worth aborting an + extraction over. It warns rather than falling back quietly, because that + fallback silently restores the very behavior this fixes, and the skill-side + caller substitutes this path by hand. + """ + memo_key = None + if prompt_file is not None: + prompt = Path(prompt_file) + try: + st = prompt.stat() + memo_key = (str(prompt), st.st_size, st.st_mtime_ns) + if memo_key in _prompt_fp_cache: + return _prompt_fp_cache[memo_key] + except OSError: + pass # unreadable — fall through to the warning below + if prompt is None: + return None + try: + fp = prompt_fingerprint(prompt) + if memo_key is not None: + _prompt_fp_cache[memo_key] = fp + return fp + except (OSError, UnicodeError) as exc: + warnings.warn( + f"could not read extraction prompt {str(prompt)!r} ({exc}); semantic cache " + "entries cannot be attributed to a prompt version and fall back to the " + "unversioned layout, so this run may replay entries from an older " + "extraction prompt (#1939).", + RuntimeWarning, + stacklevel=3, + ) + return None + + # A frontmatter delimiter is a whole line of exactly three dashes (optional # trailing whitespace). Substring checks like startswith("---") / # find("\n---") also match `----` thematic breaks and `--- text` prose, @@ -338,7 +431,8 @@ def _absolutize_source_files_in(payload: dict, root: Path) -> None: continue -def cache_dir(root: Path = Path("."), kind: str = "ast") -> Path: +def cache_dir(root: Path = Path("."), kind: str = "ast", + prompt_fp: str | None = None) -> Path: """Returns the cache directory for ``kind`` - creates it if needed. kind is "ast", "semantic", or a mode-namespaced semantic kind such as @@ -347,9 +441,14 @@ def cache_dir(root: Path = Path("."), kind: str = "ast") -> Path: AST entries live in graphify-out/cache/ast/v{version}/ — namespaced by graphify version because they depend on extractor code, not just file - contents. Semantic entries live unversioned in graphify-out/cache/semantic/ - (re-extraction costs LLM calls); deep-mode entries live beside them in - graphify-out/cache/semantic-deep/. + contents. Semantic entries are still NOT version-namespaced (re-extraction + costs LLM calls, #1252): they live in graphify-out/cache/semantic/, with + deep-mode entries beside them in graphify-out/cache/semantic-deep/. + + ``prompt_fp`` (semantic kinds only) adds a p{fingerprint}/ subdirectory so + entries are attributed to the extraction prompt that produced them (#1939). + Omitting it yields the historical flat layout, where entries of unknown + vintage live. """ _out = Path(_GRAPHIFY_OUT) base = _out if _out.is_absolute() else Path(root).resolve() / _out @@ -357,12 +456,16 @@ def cache_dir(root: Path = Path("."), kind: str = "ast") -> Path: if kind == "ast": d = d / f"v{_EXTRACTOR_VERSION}" _cleanup_stale_ast_entries(d.parent, d) + elif prompt_fp: + d = d / f"p{prompt_fp}" d.mkdir(parents=True, exist_ok=True) return d def load_cached(path: Path, root: Path = Path("."), kind: str = "ast", - cache_root: Path | None = None) -> dict | None: + cache_root: Path | None = None, prompt: "str | Path | None" = None, + prompt_file: "str | Path | None" = None, + allow_legacy: bool = True) -> dict | None: """Return cached extraction for this file if hash matches, else None. Cache key: SHA256 of file contents. @@ -380,19 +483,39 @@ def load_cached(path: Path, root: Path = Path("."), kind: str = "ast", flat cache/ layout (pre-0.5.3) and the unversioned cache/ast/ layout — are deliberately not consulted: they were produced by a different extractor and may be stale. + + ``prompt`` (semantic kinds) is the extraction prompt — text, or a Path to + the prompt file — that the caller is about to extract with. It selects the + p{fingerprint}/ namespace, so an entry produced by a different prompt is a + miss rather than a silent stale hit (#1939). When it is given and the + fingerprinted namespace misses, ``allow_legacy`` (default True) falls back + to a flat-layout entry: those predate fingerprinting, so their vintage is + unknowable — they are served rather than re-billed, and the hit is counted + so :func:`check_semantic_cache` can report N to the user. Callers that must + not mix vintages within one entry (see :func:`save_semantic_cache`'s + ``merge_existing``) pass allow_legacy=False. Returns None if no cache entry or file has changed. """ + global _legacy_semantic_hits location = cache_root if cache_root is not None else root try: h = file_hash(path, root, cache_root=cache_root) except OSError: return None - entry = cache_dir(location, kind) / f"{h}.json" + prompt_fp = _resolve_prompt_fp(prompt, prompt_file) + entry = cache_dir(location, kind, prompt_fp) / f"{h}.json" + legacy_hit = False + if prompt_fp and not entry.exists() and allow_legacy: + legacy = cache_dir(location, kind) / f"{h}.json" + if legacy.exists(): + entry, legacy_hit = legacy, True if entry.exists(): try: result = json.loads(entry.read_text(encoding="utf-8")) except (json.JSONDecodeError, OSError): return None + if legacy_hit: + _legacy_semantic_hits += 1 # Re-anchor relative source_file fields so callers see the same # absolute-path shape that a fresh in-process extraction produces # (#777). Legacy entries with absolute source_file pass through. @@ -403,7 +526,8 @@ def load_cached(path: Path, root: Path = Path("."), kind: str = "ast", def save_cached(path: Path, result: dict, root: Path = Path("."), kind: str = "ast", - cache_root: Path | None = None) -> None: + cache_root: Path | None = None, prompt: "str | Path | None" = None, + prompt_file: "str | Path | None" = None) -> None: """Save extraction result for this file. Stores as graphify-out/cache/{kind}/{hash}.json where hash = SHA256 of current file contents. @@ -413,6 +537,13 @@ def save_cached(path: Path, result: dict, root: Path = Path("."), kind: str = "a ``cache_root`` (when given) is where the cache directory is written, decoupled from ``root`` so the cache never lands inside the analyzed source tree (#1774). + ``prompt`` (semantic kinds) is the extraction prompt that produced ``result`` + — text, or a Path to the prompt file. It stamps the entry into the + p{fingerprint}/ namespace so a later run under a different prompt does not + replay it (#1939). Writes always land in the fingerprinted namespace when a + prompt is given: an entry of known vintage is never written back into the + flat unknown-vintage layout. + No-ops if `path` is not a regular file. Subagent-produced semantic fragments occasionally carry a directory path in `source_file`; skipping them prevents IsADirectoryError from aborting the whole batch. @@ -437,7 +568,7 @@ def save_cached(path: Path, result: dict, root: Path = Path("."), kind: str = "a _relativize_source_files_in(on_disk, root) h = file_hash(p, root, cache_root=cache_root) location = cache_root if cache_root is not None else root - target_dir = cache_dir(location, kind) + target_dir = cache_dir(location, kind, _resolve_prompt_fp(prompt, prompt_file)) entry = target_dir / f"{h}.json" fd, tmp_path = tempfile.mkstemp(dir=target_dir, prefix=f"{h}.", suffix=".tmp") try: @@ -470,13 +601,14 @@ def cached_files(root: Path = Path(".")) -> set[str]: # Legacy flat entries if base.is_dir(): hashes.update(p.stem for p in base.glob("*.json")) - # Namespaced entries (ast/ recursively, covering per-version subdirs; - # semantic-deep/ holds --mode deep entries, #1894) - for kind, pattern in (("ast", "**/*.json"), ("semantic", "*.json"), - ("semantic-deep", "*.json")): + # Namespaced entries, all globbed recursively: ast/ has per-version subdirs, + # semantic-deep/ holds --mode deep entries (#1894), and both semantic kinds + # have per-prompt-fingerprint subdirs alongside pre-fingerprint flat entries + # (#1939). + for kind in ("ast", "semantic", "semantic-deep"): d = base / kind if d.is_dir(): - hashes.update(p.stem for p in d.glob(pattern)) + hashes.update(p.stem for p in d.glob("**/*.json")) return hashes @@ -488,13 +620,13 @@ def clear_cache(root: Path = Path(".")) -> None: if base.is_dir(): for f in base.glob("*.json"): f.unlink() - # Namespaced entries (ast/ recursively, covering per-version subdirs; - # semantic-deep/ holds --mode deep entries, #1894) - for kind, pattern in (("ast", "**/*.json"), ("semantic", "*.json"), - ("semantic-deep", "*.json")): + # Namespaced entries, all globbed recursively: ast/ has per-version subdirs, + # semantic-deep/ holds --mode deep entries (#1894), and both semantic kinds + # have per-prompt-fingerprint subdirs (#1939). + for kind in ("ast", "semantic", "semantic-deep"): d = base / kind if d.is_dir(): - for f in d.glob(pattern): + for f in d.glob("**/*.json"): f.unlink() @@ -519,6 +651,16 @@ def prune_semantic_cache(root: Path, live_hashes: set[str]) -> int: touched (never ``cache/ast/**`` or anything else). The unversioned design is preserved: we prune by liveness, not by version. + The sweep recurses into the per-prompt-fingerprint subdirs (#1939) for the + same reason it covers the deep namespace: a glob that stopped at the top + level would leave every fingerprinted entry permanently unprunable. Entries + under a fingerprint other than the current one are pruned by liveness only, + never swept wholesale the way :func:`_cleanup_stale_ast_entries` sweeps old + AST versions — two hosts with different prompts (verbose vs compact + extraction-spec) can share one graphify-out/, and a wholesale sweep would + have each run delete the other's entries and re-bill extraction on every + alternation. Liveness keeps the total bounded by live docs × prompts seen. + Best-effort, mirroring :func:`_cleanup_stale_ast_entries`: each unlink is wrapped in ``try/except OSError`` and a failure is ignored. The worst-case failure mode is benign — a surviving orphan costs only one re-extraction of @@ -531,7 +673,7 @@ def prune_semantic_cache(root: Path, live_hashes: set[str]) -> int: semantic_dir = base / "cache" / kind if not semantic_dir.is_dir(): continue - for entry in semantic_dir.glob("*.json"): + for entry in semantic_dir.glob("**/*.json"): if entry.stem in live_hashes: continue try: @@ -546,6 +688,8 @@ def check_semantic_cache( files: list[str], root: Path = Path("."), mode: str | None = None, + prompt: "str | Path | None" = None, + prompt_file: "str | Path | None" = None, ) -> tuple[list[dict], list[dict], list[dict], list[str]]: """Check semantic extraction cache for a list of absolute file paths. @@ -558,18 +702,30 @@ def check_semantic_cache( are unaffected. A non-None mode (e.g. ``"deep"``) reads ``cache/semantic-{mode}/`` instead, so deep-mode results never shadow (or get shadowed by) standard-mode entries for the same content (#1894). + + ``prompt`` is the extraction prompt this run will use for the uncached + files — the prompt text (Python path) or a Path to the prompt file the + agent loaded (skill path, ``references/extraction-spec.md``). Supplying it + restricts hits to entries produced by that same prompt, so an upgrade that + changed the prompt re-extracts instead of replaying the older vintage + (#1939). Entries written before fingerprinting existed still hit — their + vintage is unknowable and dropping them would re-bill a whole corpus — but + a warning reports how many were served. Omitting ``prompt`` keeps the + historical behavior for existing callers. """ + global _legacy_semantic_hits kind = "semantic" if mode is None else f"semantic-{mode}" cached_nodes: list[dict] = [] cached_edges: list[dict] = [] cached_hyperedges: list[dict] = [] uncached: list[str] = [] + legacy_before = _legacy_semantic_hits for fpath in files: p = Path(fpath) if not p.is_absolute(): p = Path(root) / p - result = load_cached(p, root, kind=kind) + result = load_cached(p, root, kind=kind, prompt=prompt, prompt_file=prompt_file) if result is not None: cached_nodes.extend(result.get("nodes", [])) cached_edges.extend(result.get("edges", [])) @@ -577,6 +733,18 @@ def check_semantic_cache( else: uncached.append(fpath) + legacy = _legacy_semantic_hits - legacy_before + if legacy: + warnings.warn( + f"{legacy} semantic cache entr{'y' if legacy == 1 else 'ies'} predate " + "extraction-prompt fingerprinting and were written by an unknown prompt " + "version; they were replayed as-is, so this graph may mix extraction " + "vintages. Re-run with --force (or GRAPHIFY_FORCE=1) to re-extract them " + "with the current prompt (#1939).", + RuntimeWarning, + stacklevel=2, + ) + return cached_nodes, cached_edges, cached_hyperedges, uncached @@ -588,6 +756,8 @@ def save_semantic_cache( merge_existing: bool = False, allowed_source_files: Iterable[str | Path] | None = None, mode: str | None = None, + prompt: "str | Path | None" = None, + prompt_file: "str | Path | None" = None, ) -> int: """Save semantic extraction results to cache, keyed by source_file. @@ -611,6 +781,13 @@ def save_semantic_cache( cache-write keys. Semantic nodes can legitimately mention another corpus file, but a model must not be able to replace that file's complete cache entry unless the file was part of the current extraction batch (#1757). + + ``prompt`` is the extraction prompt that produced these results — text, or + a Path to the prompt file. It stamps entries into the p{fingerprint}/ + namespace so a later run under a different prompt re-extracts rather than + replaying them (#1939). Pass the same prompt here as to + :func:`check_semantic_cache`, or the write lands in a namespace the next + read won't consult. Returns the number of files cached. """ from collections import defaultdict @@ -716,13 +893,19 @@ def hyperedge_dangles(h: dict) -> bool: ) continue if merge_existing: - prev = load_cached(p, root, kind=kind) + # allow_legacy=False: merging a pre-fingerprint entry into this + # write would fuse two prompt vintages inside a single entry and + # then stamp the result as current-vintage — the exact mixing + # #1939 is about, made unfixable because the entry now claims a + # prompt that only produced half of it. + prev = load_cached(p, root, kind=kind, prompt=prompt, + prompt_file=prompt_file, allow_legacy=False) if prev: result = { "nodes": (prev.get("nodes", []) or []) + result["nodes"], "edges": (prev.get("edges", []) or []) + result["edges"], "hyperedges": (prev.get("hyperedges", []) or []) + result["hyperedges"], } - save_cached(p, result, root, kind=kind) + save_cached(p, result, root, kind=kind, prompt=prompt, prompt_file=prompt_file) saved += 1 return saved diff --git a/graphify/cli.py b/graphify/cli.py index d817401ac..2700f68df 100644 --- a/graphify/cli.py +++ b/graphify/cli.py @@ -2553,6 +2553,12 @@ def _parse_float(name: str, raw: str) -> float: # Deep mode uses its own namespace (cache/semantic-deep/) so deep and # standard results for the same content never shadow each other (#1894). sem_cache_mode = "deep" if deep_mode else None + # Entries are attributed to the extraction prompt that produced them, so + # a release that changes the prompt re-extracts rather than replaying the + # older vintage alongside the new one (#1939). Read and write must pass + # the same prompt, or the write lands where the next read won't look. + from graphify.llm import _extraction_system as _sem_prompt_for + sem_prompt = _sem_prompt_for(deep=deep_mode) if semantic_files: sem_paths_str = [str(p) for p in semantic_files] if force: @@ -2563,7 +2569,8 @@ def _parse_float(name: str, raw: str) -> float: uncached_paths = list(sem_paths_str) else: cached_nodes, cached_edges, cached_hyperedges, uncached_paths = ( - _check_semantic_cache(sem_paths_str, root=out_root, mode=sem_cache_mode) + _check_semantic_cache(sem_paths_str, root=out_root, mode=sem_cache_mode, + prompt=sem_prompt) ) sem_cache_hits = len(semantic_files) - len(uncached_paths) sem_cache_misses = len(uncached_paths) @@ -2636,6 +2643,7 @@ def _progress(idx: int, total: int, _result: dict) -> None: root=out_root, allowed_source_files=uncached_paths, mode=sem_cache_mode, + prompt=sem_prompt, ) except Exception as exc: print(f"[graphify extract] warning: could not write semantic cache: {exc}", file=sys.stderr) @@ -2947,20 +2955,27 @@ def _progress(idx: int, total: int, _result: dict) -> None: elif cmd == "cache-check": # graphify cache-check [--root ] [--mode | --deep] + # [--prompt-file ] # Reads file paths (one per line) from , checks semantic cache. # --mode deep (or --deep) checks the cache/semantic-deep/ namespace # written by `extract --mode deep` instead of cache/semantic/ (#1894). + # --prompt-file names the extraction prompt the caller will use (an agent's + # references/extraction-spec.md), restricting hits to entries produced by + # that same prompt (#1939). Omitting it reads the unattributed layout, which + # cannot see entries a fingerprinted run wrote. # Writes: # graphify-out/.graphify_cached.json — already-cached nodes/edges/hyperedges # graphify-out/.graphify_uncached.txt — paths that need extraction # Stdout: "Cache: N hit, M miss" from graphify.cache import check_semantic_cache if len(sys.argv) < 3: - print("Usage: graphify cache-check [--root ] [--mode | --deep]", file=sys.stderr) + print("Usage: graphify cache-check [--root ] " + "[--mode | --deep] [--prompt-file ]", file=sys.stderr) sys.exit(1) files_from = Path(sys.argv[2]) root = Path(".") cache_mode: str | None = None + prompt_file: str | None = None i = 3 while i < len(sys.argv): if sys.argv[i] == "--root" and i + 1 < len(sys.argv): @@ -2975,11 +2990,17 @@ def _progress(idx: int, total: int, _result: dict) -> None: elif sys.argv[i] == "--deep": cache_mode = "deep" i += 1 + elif sys.argv[i] == "--prompt-file" and i + 1 < len(sys.argv): + prompt_file = sys.argv[i + 1] + i += 2 + elif sys.argv[i].startswith("--prompt-file="): + prompt_file = sys.argv[i].split("=", 1)[1] + i += 1 else: i += 1 files = [f for f in files_from.read_text(encoding="utf-8").splitlines() if f.strip()] cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache( - files, root, mode=cache_mode + files, root, mode=cache_mode, prompt_file=prompt_file ) out = root / _GRAPHIFY_OUT out.mkdir(parents=True, exist_ok=True) diff --git a/graphify/llm.py b/graphify/llm.py index e3dbf681b..3ce10f283 100644 --- a/graphify/llm.py +++ b/graphify/llm.py @@ -1935,6 +1935,10 @@ def _checkpoint_chunk(result: dict, chunk: "list[Path | FileSlice]") -> None: merge_existing=True, allowed_source_files=allowed, mode="deep" if deep_mode else None, + # Stamp the entry with the prompt that produced it, so a release + # that changes _EXTRACTION_SYSTEM re-extracts instead of replaying + # this vintage forever (#1939). + prompt=_extraction_system(deep=deep_mode), ) except Exception as _exc: # noqa: BLE001 — checkpoint is best-effort print(f"[graphify] incremental cache checkpoint failed: {_exc}", file=sys.stderr) diff --git a/graphify/skill-agents.md b/graphify/skill-agents.md index 4969929c0..19d5bea4e 100644 --- a/graphify/skill-agents.md +++ b/graphify/skill-agents.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -311,7 +313,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/graphify/skill-amp.md b/graphify/skill-amp.md index 4969929c0..19d5bea4e 100644 --- a/graphify/skill-amp.md +++ b/graphify/skill-amp.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -311,7 +313,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/graphify/skill-claw.md b/graphify/skill-claw.md index bafbca074..bf9dd3431 100644 --- a/graphify/skill-claw.md +++ b/graphify/skill-claw.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -314,7 +316,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/graphify/skill-codex.md b/graphify/skill-codex.md index d71f4592d..04eb5705b 100644 --- a/graphify/skill-codex.md +++ b/graphify/skill-codex.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -311,7 +313,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/graphify/skill-copilot.md b/graphify/skill-copilot.md index bafbca074..bf9dd3431 100644 --- a/graphify/skill-copilot.md +++ b/graphify/skill-copilot.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -314,7 +316,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/graphify/skill-droid.md b/graphify/skill-droid.md index ae1d49325..ffa94995d 100644 --- a/graphify/skill-droid.md +++ b/graphify/skill-droid.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -311,7 +313,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/graphify/skill-kilo.md b/graphify/skill-kilo.md index 6aa87a8bd..96dfc2e85 100644 --- a/graphify/skill-kilo.md +++ b/graphify/skill-kilo.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -314,7 +316,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/graphify/skill-kiro.md b/graphify/skill-kiro.md index bafbca074..bf9dd3431 100644 --- a/graphify/skill-kiro.md +++ b/graphify/skill-kiro.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -314,7 +316,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/graphify/skill-opencode.md b/graphify/skill-opencode.md index 8c3ce3eed..6613434e8 100644 --- a/graphify/skill-opencode.md +++ b/graphify/skill-opencode.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -297,7 +299,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -306,7 +308,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/graphify/skill-pi.md b/graphify/skill-pi.md index bafbca074..bf9dd3431 100644 --- a/graphify/skill-pi.md +++ b/graphify/skill-pi.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -314,7 +316,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/graphify/skill-trae.md b/graphify/skill-trae.md index af59166ae..407f091b6 100644 --- a/graphify/skill-trae.md +++ b/graphify/skill-trae.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -303,7 +305,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -312,7 +314,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/graphify/skill-vscode.md b/graphify/skill-vscode.md index 11a2837b5..f7e7f8442 100644 --- a/graphify/skill-vscode.md +++ b/graphify/skill-vscode.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -301,7 +303,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -310,7 +312,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/graphify/skill-windows.md b/graphify/skill-windows.md index 0dc73947a..9c7ab7420 100644 --- a/graphify/skill-windows.md +++ b/graphify/skill-windows.md @@ -236,6 +236,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -248,7 +250,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -327,7 +329,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -336,7 +338,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/graphify/skill.md b/graphify/skill.md index bafbca074..bf9dd3431 100644 --- a/graphify/skill.md +++ b/graphify/skill.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -314,7 +316,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tests/test_cache.py b/tests/test_cache.py index ff89a3261..993c5a0d2 100644 --- a/tests/test_cache.py +++ b/tests/test_cache.py @@ -903,3 +903,234 @@ def test_save_semantic_cache_merge_existing_prunes_only_incoming(tmp_path): assert ("a", "a") in pairs, "prior entry's valid edge must survive the union" assert ("a", "b") in pairs, "incoming valid edge must be kept" assert not any("stray" in p for p in pairs) + + +# --- extraction-prompt fingerprinting (#1939) ------------------------------- + + +def test_prompt_fingerprint_stable_and_prompt_sensitive(tmp_path): + """The fingerprint is stable for identical prompts and differs when the + prompt text changes — the whole invalidation signal rests on this.""" + from graphify.cache import prompt_fingerprint + + assert prompt_fingerprint("extract a graph") == prompt_fingerprint("extract a graph") + assert prompt_fingerprint("extract a graph") != prompt_fingerprint("extract a graph v2") + + # A Path is read and hashed as its contents, so the skill path (which loads + # references/extraction-spec.md) and the Python path agree on the same text. + spec = tmp_path / "extraction-spec.md" + spec.write_text("extract a graph", encoding="utf-8") + assert prompt_fingerprint(spec) == prompt_fingerprint("extract a graph") + + +def test_prompt_fingerprint_ignores_line_endings(tmp_path): + """A CRLF checkout of the same spec must not look like a prompt change — + otherwise every Windows run re-bills the whole corpus.""" + from graphify.cache import prompt_fingerprint + + assert prompt_fingerprint("a\r\nb\r\n") == prompt_fingerprint("a\nb\n") + assert prompt_fingerprint("a \nb\n") == prompt_fingerprint("a\nb\n") + + +def test_semantic_cache_prompt_change_invalidates(tmp_path): + """The reported bug (#1939): after the extraction prompt changes, an + unchanged file must MISS instead of replaying the older vintage.""" + from graphify.cache import check_semantic_cache, save_semantic_cache + + f = tmp_path / "doc.md" + f.write_text("# Doc\n\nBody.\n") + save_semantic_cache([{"id": "old_vintage", "source_file": "doc.md"}], [], + root=tmp_path, prompt="PROMPT V1") + + # Same prompt: hit. + nodes, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V1") + assert [n["id"] for n in nodes] == ["old_vintage"] + assert uncached == [] + + # Prompt changed (an upgrade shipped a new extraction-spec): must re-extract. + nodes, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V2") + assert nodes == [] + assert uncached == [str(f)], "a new prompt must not replay the old prompt's entry" + + # V2's results land in their own namespace and do not clobber V1's, so + # rolling back to V1 still hits rather than re-billing. + save_semantic_cache([{"id": "new_vintage", "source_file": "doc.md"}], [], + root=tmp_path, prompt="PROMPT V2") + nodes, _, _, _ = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V2") + assert [n["id"] for n in nodes] == ["new_vintage"] + nodes, _, _, _ = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V1") + assert [n["id"] for n in nodes] == ["old_vintage"] + + +def test_semantic_cache_prompt_namespaced_layout(tmp_path): + """Fingerprinted entries live under cache/semantic/p{fp}/, never flat.""" + from graphify.cache import prompt_fingerprint, save_semantic_cache + + f = tmp_path / "doc.md" + f.write_text("# Doc\n") + save_semantic_cache([{"id": "n", "source_file": "doc.md"}], [], + root=tmp_path, prompt="PROMPT V1") + + sem = tmp_path / "graphify-out" / "cache" / "semantic" + h = file_hash(f, tmp_path) + assert (sem / f"p{prompt_fingerprint('PROMPT V1')}" / f"{h}.json").exists() + assert not (sem / f"{h}.json").exists(), ( + "a known-vintage entry must never be written into the flat unknown-vintage layout" + ) + + +def test_semantic_cache_prompt_and_mode_compose(tmp_path): + """The prompt fingerprint nests inside the deep namespace (#1894), so the + two dimensions are independent.""" + from graphify.cache import check_semantic_cache, save_semantic_cache + + f = tmp_path / "doc.md" + f.write_text("# Doc\n") + save_semantic_cache([{"id": "d", "source_file": "doc.md"}], [], + root=tmp_path, mode="deep", prompt="PROMPT V1") + + deep = tmp_path / "graphify-out" / "cache" / "semantic-deep" + assert list(deep.glob("p*/*.json")), "deep + prompt must nest under semantic-deep/p{fp}/" + + # Right mode, wrong prompt -> miss. Right prompt, wrong mode -> miss. + _, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, mode="deep", + prompt="PROMPT V2") + assert uncached == [str(f)] + _, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, prompt="PROMPT V1") + assert uncached == [str(f)] + # Both right -> hit. + nodes, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, mode="deep", + prompt="PROMPT V1") + assert [n["id"] for n in nodes] == ["d"] and uncached == [] + + +def test_semantic_cache_legacy_entries_served_with_warning(tmp_path): + """Entries written before fingerprinting have unknowable vintage. They are + still served — dropping them would re-bill a whole corpus on upgrade — but + the user is told how many, which is the signal #1939 says is missing today.""" + from graphify.cache import check_semantic_cache, save_semantic_cache + + a = tmp_path / "a.md" + a.write_text("# A\n") + b = tmp_path / "b.md" + b.write_text("# B\n") + # Pre-fingerprint writes: the historical flat layout. + save_semantic_cache([{"id": "a_old", "source_file": "a.md"}, + {"id": "b_old", "source_file": "b.md"}], [], root=tmp_path) + + with pytest.warns(RuntimeWarning, match="2 semantic cache entries predate"): + nodes, _, _, uncached = check_semantic_cache( + [str(a), str(b)], root=tmp_path, prompt="PROMPT V1" + ) + assert {n["id"] for n in nodes} == {"a_old", "b_old"} + assert uncached == [] + + +def test_semantic_cache_fingerprinted_entry_beats_legacy(tmp_path): + """Once a file is re-extracted under the current prompt, its fingerprinted + entry wins and the stale flat one is no longer consulted (no warning).""" + import warnings as _warnings + from graphify.cache import check_semantic_cache, save_semantic_cache + + f = tmp_path / "doc.md" + f.write_text("# Doc\n") + save_semantic_cache([{"id": "unknown_vintage", "source_file": "doc.md"}], [], + root=tmp_path) # legacy flat + save_semantic_cache([{"id": "current", "source_file": "doc.md"}], [], + root=tmp_path, prompt="PROMPT V1") + + with _warnings.catch_warnings(): + _warnings.simplefilter("error") # any legacy warning would raise here + nodes, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, + prompt="PROMPT V1") + assert [n["id"] for n in nodes] == ["current"] + assert uncached == [] + + +def test_semantic_cache_merge_existing_never_fuses_legacy_vintage(tmp_path): + """merge_existing must not union a pre-fingerprint entry into a write it is + about to stamp as current-vintage — that would mix two prompts inside one + entry and then attest the result to a prompt that produced half of it.""" + from graphify.cache import load_cached, save_semantic_cache + + f = tmp_path / "doc.md" + f.write_text("# Doc\n") + save_semantic_cache([{"id": "unknown_vintage", "source_file": "doc.md"}], [], + root=tmp_path) # legacy flat + save_semantic_cache([{"id": "current", "source_file": "doc.md"}], [], + root=tmp_path, merge_existing=True, prompt="PROMPT V1") + + entry = load_cached(f, root=tmp_path, kind="semantic", prompt="PROMPT V1") + assert [n["id"] for n in entry["nodes"]] == ["current"] + + # Within one prompt, merge_existing still unions across checkpoints. + save_semantic_cache([{"id": "second_chunk", "source_file": "doc.md"}], [], + root=tmp_path, merge_existing=True, prompt="PROMPT V1") + entry = load_cached(f, root=tmp_path, kind="semantic", prompt="PROMPT V1") + assert {n["id"] for n in entry["nodes"]} == {"current", "second_chunk"} + + +def test_semantic_prune_and_clear_reach_fingerprint_subdirs(tmp_path): + """A glob that stopped at the top level would leave every fingerprinted + entry unprunable, re-growing the unbounded-orphan problem of #1527.""" + from graphify.cache import ( + cached_files, clear_cache, prune_semantic_cache, save_semantic_cache, + ) + + f = tmp_path / "doc.md" + f.write_text("# Doc\n") + save_semantic_cache([{"id": "n", "source_file": "doc.md"}], [], + root=tmp_path, prompt="PROMPT V1") + h = file_hash(f, tmp_path) + assert h in cached_files(tmp_path), "cached_files must see fingerprinted entries" + + # Live: kept. + assert prune_semantic_cache(tmp_path, {h}) == 0 + # Orphaned (content changed / file deleted): pruned. + assert prune_semantic_cache(tmp_path, set()) == 1 + + save_semantic_cache([{"id": "n", "source_file": "doc.md"}], [], + root=tmp_path, prompt="PROMPT V1") + clear_cache(tmp_path) + assert not list((tmp_path / "graphify-out" / "cache" / "semantic").glob("**/*.json")) + + +def test_semantic_cache_unreadable_prompt_file_warns_and_falls_back(tmp_path): + """A skill snippet substitutes SPEC_PATH by hand. If it lands on a path that + isn't there, the fallback to the unattributed layout must be loud: silently + reverting to unversioned keying is exactly the #1939 behavior being fixed.""" + from graphify.cache import check_semantic_cache, save_semantic_cache + + f = tmp_path / "doc.md" + f.write_text("# Doc\n") + save_semantic_cache([{"id": "n", "source_file": "doc.md"}], [], root=tmp_path) + + with pytest.warns(RuntimeWarning, match="could not read extraction prompt"): + nodes, _, _, uncached = check_semantic_cache( + [str(f)], root=tmp_path, prompt_file=str(tmp_path / "nope.md") + ) + # Fell back rather than aborting the run. + assert [n["id"] for n in nodes] == ["n"] and uncached == [] + + +def test_prompt_file_reflects_edited_spec(tmp_path): + """The prompt-file fingerprint is memoized per (path, size, mtime); an edited + spec must still register as a new prompt rather than reusing a stale memo.""" + from graphify.cache import check_semantic_cache, save_semantic_cache + + spec = tmp_path / "extraction-spec.md" + spec.write_text("prompt one", encoding="utf-8") + f = tmp_path / "doc.md" + f.write_text("# Doc\n") + + save_semantic_cache([{"id": "v1", "source_file": "doc.md"}], [], + root=tmp_path, prompt_file=str(spec)) + nodes, _, _, _ = check_semantic_cache([str(f)], root=tmp_path, prompt_file=str(spec)) + assert [n["id"] for n in nodes] == ["v1"] + + # An upgrade rewrites the spec: the same file path is now a different prompt. + import os as _os + spec.write_text("prompt two — rewritten by an upgrade", encoding="utf-8") + _os.utime(spec, ns=(0, 0)) # force a distinct stat signature + _, _, _, uncached = check_semantic_cache([str(f)], root=tmp_path, prompt_file=str(spec)) + assert uncached == [str(f)], "an edited spec must invalidate, not reuse the memo" diff --git a/tests/test_chunking.py b/tests/test_chunking.py index a978a5daa..19e65efbc 100644 --- a/tests/test_chunking.py +++ b/tests/test_chunking.py @@ -321,8 +321,11 @@ def stray(chunk, **kwargs): assert [n["id"] for n in after["nodes"]] == ["b_real"], ( f"B.py cache was clobbered by an out-of-chunk node: {after}" ) - # A.py (the actual chunk file) was legitimately cached. - a_cache = load_cached(a, tmp_path, kind="semantic") + # A.py (the actual chunk file) was legitimately cached. The checkpoint stamps + # entries with the prompt that produced them (#1939), so read that namespace. + from graphify.llm import _extraction_system + + a_cache = load_cached(a, tmp_path, kind="semantic", prompt=_extraction_system()) assert a_cache and any(n["id"] == "a_ok" for n in a_cache["nodes"]) @@ -330,7 +333,7 @@ def test_checkpoint_writes_deep_namespace_in_deep_mode(tmp_path): """#1894: the per-chunk checkpoint must follow the run's mode — a deep_mode=True run checkpoints into cache/semantic-deep/, leaving the standard cache/semantic/ namespace untouched (and vice versa).""" - from graphify.llm import extract_corpus_parallel + from graphify.llm import extract_corpus_parallel, _extraction_system from graphify.cache import load_cached doc = tmp_path / "doc.md" @@ -349,11 +352,15 @@ def ok(chunk, **kwargs): deep_mode=True, ) - deep = load_cached(doc, tmp_path, kind="semantic-deep") + # The checkpoint also stamps entries with the prompt that produced them + # (#1939) — a deep run's prompt carries the deep suffix. + deep = load_cached(doc, tmp_path, kind="semantic-deep", + prompt=_extraction_system(deep=True)) assert deep and [n["id"] for n in deep["nodes"]] == ["d1"], ( "deep-mode checkpoint must land in cache/semantic-deep/" ) - assert load_cached(doc, tmp_path, kind="semantic") is None, ( + assert load_cached(doc, tmp_path, kind="semantic", + prompt=_extraction_system(deep=False)) is None, ( "deep-mode checkpoint must not write the standard semantic namespace" ) @@ -484,7 +491,9 @@ def test_checkpoint_caches_sliced_document_chunks(tmp_path, capsys): split into FileSlice units; before the fix each sliced chunk leaked the FileSlice object into the allowlist, so save_semantic_cache raised TypeError, the best-effort except swallowed it, and the slice was never checkpointed.""" - from graphify.llm import extract_corpus_parallel, expand_oversized_files, _FILE_CHAR_CAP + from graphify.llm import ( + extract_corpus_parallel, expand_oversized_files, _FILE_CHAR_CAP, _extraction_system, + ) from graphify.file_slice import FileSlice from graphify.cache import load_cached @@ -510,7 +519,8 @@ def sliced(chunk, **kwargs): assert "incremental cache checkpoint failed" not in capsys.readouterr().err, ( "checkpoint raised on a FileSlice chunk (#1870)" ) - cached = load_cached(doc, tmp_path, kind="semantic") + # The checkpoint stamps entries with the prompt that produced them (#1939). + cached = load_cached(doc, tmp_path, kind="semantic", prompt=_extraction_system()) assert cached and any(n["id"] == "big_title" for n in cached["nodes"]), ( "sliced document was never checkpointed (#1870)" ) diff --git a/tests/test_extract_cli.py b/tests/test_extract_cli.py index 87a65e45e..d18bf5ab8 100644 --- a/tests/test_extract_cli.py +++ b/tests/test_extract_cli.py @@ -1,6 +1,8 @@ """Tests for `graphify extract` CLI dispatch path in graphify.__main__.""" from __future__ import annotations +import os + import pytest import graphify.__main__ as mainmod @@ -360,8 +362,9 @@ def test_extract_mode_deep_dispatches_over_warm_cache(monkeypatch, tmp_path): assert len(calls) == 2, ( "second deep run must be served from cache/semantic-deep/" ) - # The deep entry landed in its own namespace, not cache/semantic/. - assert any((corpus / "graphify-out" / "cache" / "semantic-deep").glob("*.json")) + # The deep entry landed in its own namespace, not cache/semantic/. Entries are + # nested under a p{prompt-fingerprint}/ subdir (#1939), hence the recursive glob. + assert any((corpus / "graphify-out" / "cache" / "semantic-deep").glob("**/*.json")) def test_extract_force_flag_redispatches_and_stamps_manifest(monkeypatch, tmp_path): @@ -393,7 +396,8 @@ def test_extract_force_flag_redispatches_and_stamps_manifest(monkeypatch, tmp_pa assert calls[1]["paths"] == [str(corpus / "README.md")] # The forced run still wrote the semantic cache and stamped the manifest. - assert any((corpus / "graphify-out" / "cache" / "semantic").glob("*.json")) + # Entries nest under a p{prompt-fingerprint}/ subdir (#1939). + assert any((corpus / "graphify-out" / "cache" / "semantic").glob("**/*.json")) manifest = json.loads( (corpus / "graphify-out" / "manifest.json").read_text() ) @@ -824,3 +828,30 @@ def test_no_cluster_incremental_prunes_newly_excluded_file( f"--no-cluster early exit must prune excluded sources, still see {sources}" ) assert any("keep.py" in s for s in sources) + + +def test_cache_check_prompt_file_scopes_hits_to_that_prompt(monkeypatch, tmp_path, capsys): + """#1939: cache-check --prompt-file only counts entries produced by that same + extraction prompt, so an upgraded prompt reports a miss (re-extract) rather + than replaying the older vintage.""" + from graphify.cache import save_semantic_cache + + doc = tmp_path / "doc.md" + doc.write_text("# Doc\n") + spec = tmp_path / "extraction-spec.md" + spec.write_text("PROMPT V1", encoding="utf-8") + save_semantic_cache([{"id": "d", "source_file": "doc.md"}], [], + root=tmp_path, prompt_file=str(spec)) + files_from = tmp_path / "files.txt" + files_from.write_text(str(doc) + "\n") + monkeypatch.setattr(mainmod, "_check_skill_version", lambda _: None) + + base = ["graphify", "cache-check", str(files_from), "--root", str(tmp_path)] + _run_extract(monkeypatch, base + ["--prompt-file", str(spec)]) + assert "Cache: 1 hit, 0 miss" in capsys.readouterr().out + + # An upgrade rewrites the prompt: the entry must no longer satisfy the run. + spec.write_text("PROMPT V2 — rewritten by an upgrade", encoding="utf-8") + os.utime(spec, ns=(0, 0)) + _run_extract(monkeypatch, base + ["--prompt-file", str(spec)]) + assert "Cache: 0 hit, 1 miss" in capsys.readouterr().out diff --git a/tests/test_skillgen.py b/tests/test_skillgen.py index 9797267a9..0c09e601e 100644 --- a/tests/test_skillgen.py +++ b/tests/test_skillgen.py @@ -949,3 +949,31 @@ def test_agents_audit_baseline_is_amps_v8_body(): assert gen._v8_baseline_ref("agents") == "47042beb05d1f6dd2186c0c499ae2840ce604ead:graphify/skill-amp.md" problems = gen.audit_coverage(platforms["agents"]) assert problems == [], "\n".join(problems) + + +def test_semantic_cache_calls_pass_prompt_file_for_every_split_host(): + """#1939: a skill's cache read and write must both name the extraction prompt + they use, or the run replays entries produced by an older prompt (the read) / + strands its results where the next read won't look (the write). + + Locked per host because the two calls live ~80 lines apart in the rendered + body: adding the argument to one and not the other silently disables the + cache rather than failing loudly. The monolith hosts (aider, devin) inline + their prompt instead of shipping references/extraction-spec.md and are + deliberately excluded — they have no spec path to point at. + """ + platforms = gen.load_platforms() + arts = gen.render_all(platforms) + bodies = [a for a in arts + if "check_semantic_cache(" in a.content + and "references/extraction-spec.md" in a.content] + assert bodies, "no rendered split-host skill body calls check_semantic_cache" + for a in bodies: + for call in ("check_semantic_cache(", "save_semantic_cache("): + line = next(ln for ln in a.content.splitlines() if call in ln and "import" not in ln) + assert "prompt_file='SPEC_PATH'" in line, ( + f"{a.path}: {call} must pass prompt_file so entries are attributed " + f"to the extraction prompt (#1939) — got: {line.strip()}" + ) + # The placeholder is inert unless the body tells the agent what to substitute. + assert "SPEC_PATH below is the **absolute** path" in a.content, a.path diff --git a/tools/skillgen/expected/graphify__skill-agents.md b/tools/skillgen/expected/graphify__skill-agents.md index 4969929c0..19d5bea4e 100644 --- a/tools/skillgen/expected/graphify__skill-agents.md +++ b/tools/skillgen/expected/graphify__skill-agents.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -311,7 +313,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tools/skillgen/expected/graphify__skill-amp.md b/tools/skillgen/expected/graphify__skill-amp.md index 4969929c0..19d5bea4e 100644 --- a/tools/skillgen/expected/graphify__skill-amp.md +++ b/tools/skillgen/expected/graphify__skill-amp.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -311,7 +313,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tools/skillgen/expected/graphify__skill-claw.md b/tools/skillgen/expected/graphify__skill-claw.md index bafbca074..bf9dd3431 100644 --- a/tools/skillgen/expected/graphify__skill-claw.md +++ b/tools/skillgen/expected/graphify__skill-claw.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -314,7 +316,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tools/skillgen/expected/graphify__skill-codex.md b/tools/skillgen/expected/graphify__skill-codex.md index d71f4592d..04eb5705b 100644 --- a/tools/skillgen/expected/graphify__skill-codex.md +++ b/tools/skillgen/expected/graphify__skill-codex.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -311,7 +313,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tools/skillgen/expected/graphify__skill-copilot.md b/tools/skillgen/expected/graphify__skill-copilot.md index bafbca074..bf9dd3431 100644 --- a/tools/skillgen/expected/graphify__skill-copilot.md +++ b/tools/skillgen/expected/graphify__skill-copilot.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -314,7 +316,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tools/skillgen/expected/graphify__skill-droid.md b/tools/skillgen/expected/graphify__skill-droid.md index ae1d49325..ffa94995d 100644 --- a/tools/skillgen/expected/graphify__skill-droid.md +++ b/tools/skillgen/expected/graphify__skill-droid.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -311,7 +313,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tools/skillgen/expected/graphify__skill-kilo.md b/tools/skillgen/expected/graphify__skill-kilo.md index 6aa87a8bd..96dfc2e85 100644 --- a/tools/skillgen/expected/graphify__skill-kilo.md +++ b/tools/skillgen/expected/graphify__skill-kilo.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -314,7 +316,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tools/skillgen/expected/graphify__skill-kiro.md b/tools/skillgen/expected/graphify__skill-kiro.md index bafbca074..bf9dd3431 100644 --- a/tools/skillgen/expected/graphify__skill-kiro.md +++ b/tools/skillgen/expected/graphify__skill-kiro.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -314,7 +316,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tools/skillgen/expected/graphify__skill-opencode.md b/tools/skillgen/expected/graphify__skill-opencode.md index 8c3ce3eed..6613434e8 100644 --- a/tools/skillgen/expected/graphify__skill-opencode.md +++ b/tools/skillgen/expected/graphify__skill-opencode.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -297,7 +299,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -306,7 +308,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tools/skillgen/expected/graphify__skill-pi.md b/tools/skillgen/expected/graphify__skill-pi.md index bafbca074..bf9dd3431 100644 --- a/tools/skillgen/expected/graphify__skill-pi.md +++ b/tools/skillgen/expected/graphify__skill-pi.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -314,7 +316,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tools/skillgen/expected/graphify__skill-trae.md b/tools/skillgen/expected/graphify__skill-trae.md index af59166ae..407f091b6 100644 --- a/tools/skillgen/expected/graphify__skill-trae.md +++ b/tools/skillgen/expected/graphify__skill-trae.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -303,7 +305,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -312,7 +314,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tools/skillgen/expected/graphify__skill-vscode.md b/tools/skillgen/expected/graphify__skill-vscode.md index 11a2837b5..f7e7f8442 100644 --- a/tools/skillgen/expected/graphify__skill-vscode.md +++ b/tools/skillgen/expected/graphify__skill-vscode.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -301,7 +303,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -310,7 +312,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tools/skillgen/expected/graphify__skill-windows.md b/tools/skillgen/expected/graphify__skill-windows.md index 0dc73947a..9c7ab7420 100644 --- a/tools/skillgen/expected/graphify__skill-windows.md +++ b/tools/skillgen/expected/graphify__skill-windows.md @@ -236,6 +236,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -248,7 +250,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -327,7 +329,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -336,7 +338,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tools/skillgen/expected/graphify__skill.md b/tools/skillgen/expected/graphify__skill.md index bafbca074..bf9dd3431 100644 --- a/tools/skillgen/expected/graphify__skill.md +++ b/tools/skillgen/expected/graphify__skill.md @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -314,7 +316,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ``` diff --git a/tools/skillgen/fragments/core/core.md b/tools/skillgen/fragments/core/core.md index a5e117b5c..cf8f03062 100644 --- a/tools/skillgen/fragments/core/core.md +++ b/tools/skillgen/fragments/core/core.md @@ -173,6 +173,8 @@ Before dispatching subagents, print a timing estimate: Before dispatching any subagents, check which files already have cached extraction results: +SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument. + ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -185,7 +187,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin # every source file (#1392). Video is transcribed to a document in Step 2.5 first. all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])] -cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH') +cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH') # Always (re)write the cache file: write hits, else DELETE any leftover from a prior # run so Part C never merges a stale .graphify_cached.json (#1392). @@ -240,7 +242,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens' " ``` -Save new results to cache: +Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939): ```bash $(cat graphify-out/.graphify_python) -c " import json @@ -249,7 +251,7 @@ from pathlib import Path new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]} uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line] -saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached) +saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH') print(f'Cached {saved} files') " ```