Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Full release notes with details on each version: [GitHub Releases](https://githu

## 0.9.17 (unreleased)

- Fix: the semantic cache no longer replays extractions from an older prompt after an upgrade (#1939, thanks @HunterMcGrew). Entries were keyed on `sha256(file content + path)` alone, with no component for the extraction prompt that produced them, so a release that changed the prompt left every unchanged file a cache hit: the run exited 0, `cost.json` looked cheap, and the graph silently carried two prompt generations side by side. Semantic entries are now namespaced by a fingerprint of the extraction prompt (`cache/semantic/p{fingerprint}/`, mirroring the AST cache's `v{version}/` layout), keeping both properties #1252 wanted — entries survive releases that don't touch the prompt, and invalidate only when it actually changed. The fingerprint normalizes line endings so a CRLF checkout doesn't look like a prompt change. Both extraction paths pass their prompt: the Python/CLI path (`llm.py`'s `_EXTRACTION_SYSTEM`, all backends) automatically, and the skill path via a new `prompt_file` argument in Step B0/B3 pointing at the `references/extraction-spec.md` the subagents were handed. Pre-existing entries predate fingerprinting and have unknowable vintage: they are still served rather than re-billing a whole corpus, but `check_semantic_cache` now warns with the count, so the "no signal at all" the report describes becomes a visible one; `--force` (or `GRAPHIFY_FORCE=1`) re-extracts them. Old-fingerprint entries are pruned by liveness only, never swept wholesale the way stale AST versions are — two hosts with different prompts can share one `graphify-out/`, and a wholesale sweep would have each run delete the other's entries. (The two monolith skills, aider and devin, inline their prompt instead of shipping a spec sidecar and stay on the unfingerprinted path for now.)
- Fix: a missing `manifest.json` no longer degrades `graphify extract --code-only` into a full scan that discards the committed semantic layer (#1925). On a fresh clone (or when the manifest is deliberately untracked because its mtimes churn), the incremental gate required both `manifest.json` and `graph.json`; with only the graph present it fell to a full scan, and under `--code-only` that dropped every doc/paper/image node — silently replacing a curated graph with an AST-only skeleton. An existing `graph.json` is now a sufficient incremental baseline: `detect_incremental` already treats an absent manifest as "everything new / nothing deleted", so `build_merge` + `_stale_graph_sources` preserve files that are merely out of this run's scope while still evicting genuinely deleted sources.
- Fix: hyperedge-only documents are now stamped in the manifest instead of being re-extracted on every run (#1920). `_stamped_manifest_files` (#1897) decided a semantic doc "produced output" by inspecting only `nodes` and `edges`, never `hyperedges`, so a chunk whose only output for a doc was a hyperedge (3+ nodes sharing a concept) left that doc unstamped and perpetually re-queued. Stamping now counts hyperedge output too, mirroring the per-`source_file` keying the semantic cache already uses.
- Fix: the PHP extractor now disambiguates same-named classes across namespaces (#1923). A bare class reference (`extends Page`) collapsed onto the only internal class named `Page`, so `App\Models\Page` and an imported `Filament\Pages\Page` fused into one node, manufacturing a false `inherits`/`imports` edge and a bogus cross-community bridge. A new namespace/`use`-aware resolution pass (mirroring the Java resolver, running before the unique-name rewire) re-points supertype/import references to the real definition, or parks provably-external ones on a fully-qualified stub the bare-name rewire cannot collapse. Plain, non-namespaced PHP is unchanged.
Expand Down
227 changes: 205 additions & 22 deletions graphify/cache.py

Large diffs are not rendered by default.

27 changes: 24 additions & 3 deletions graphify/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -2553,6 +2553,12 @@ def _parse_float(name: str, raw: str) -> float:
# Deep mode uses its own namespace (cache/semantic-deep/) so deep and
# standard results for the same content never shadow each other (#1894).
sem_cache_mode = "deep" if deep_mode else None
# Entries are attributed to the extraction prompt that produced them, so
# a release that changes the prompt re-extracts rather than replaying the
# older vintage alongside the new one (#1939). Read and write must pass
# the same prompt, or the write lands where the next read won't look.
from graphify.llm import _extraction_system as _sem_prompt_for
sem_prompt = _sem_prompt_for(deep=deep_mode)
if semantic_files:
sem_paths_str = [str(p) for p in semantic_files]
if force:
Expand All @@ -2563,7 +2569,8 @@ def _parse_float(name: str, raw: str) -> float:
uncached_paths = list(sem_paths_str)
else:
cached_nodes, cached_edges, cached_hyperedges, uncached_paths = (
_check_semantic_cache(sem_paths_str, root=out_root, mode=sem_cache_mode)
_check_semantic_cache(sem_paths_str, root=out_root, mode=sem_cache_mode,
prompt=sem_prompt)
)
sem_cache_hits = len(semantic_files) - len(uncached_paths)
sem_cache_misses = len(uncached_paths)
Expand Down Expand Up @@ -2636,6 +2643,7 @@ def _progress(idx: int, total: int, _result: dict) -> None:
root=out_root,
allowed_source_files=uncached_paths,
mode=sem_cache_mode,
prompt=sem_prompt,
)
except Exception as exc:
print(f"[graphify extract] warning: could not write semantic cache: {exc}", file=sys.stderr)
Expand Down Expand Up @@ -2947,20 +2955,27 @@ def _progress(idx: int, total: int, _result: dict) -> None:

elif cmd == "cache-check":
# graphify cache-check <files_from> [--root <dir>] [--mode <m> | --deep]
# [--prompt-file <path>]
# Reads file paths (one per line) from <files_from>, checks semantic cache.
# --mode deep (or --deep) checks the cache/semantic-deep/ namespace
# written by `extract --mode deep` instead of cache/semantic/ (#1894).
# --prompt-file names the extraction prompt the caller will use (an agent's
# references/extraction-spec.md), restricting hits to entries produced by
# that same prompt (#1939). Omitting it reads the unattributed layout, which
# cannot see entries a fingerprinted run wrote.
# Writes:
# graphify-out/.graphify_cached.json — already-cached nodes/edges/hyperedges
# graphify-out/.graphify_uncached.txt — paths that need extraction
# Stdout: "Cache: N hit, M miss"
from graphify.cache import check_semantic_cache
if len(sys.argv) < 3:
print("Usage: graphify cache-check <files_from> [--root <dir>] [--mode <m> | --deep]", file=sys.stderr)
print("Usage: graphify cache-check <files_from> [--root <dir>] "
"[--mode <m> | --deep] [--prompt-file <path>]", file=sys.stderr)
sys.exit(1)
files_from = Path(sys.argv[2])
root = Path(".")
cache_mode: str | None = None
prompt_file: str | None = None
i = 3
while i < len(sys.argv):
if sys.argv[i] == "--root" and i + 1 < len(sys.argv):
Expand All @@ -2975,11 +2990,17 @@ def _progress(idx: int, total: int, _result: dict) -> None:
elif sys.argv[i] == "--deep":
cache_mode = "deep"
i += 1
elif sys.argv[i] == "--prompt-file" and i + 1 < len(sys.argv):
prompt_file = sys.argv[i + 1]
i += 2
elif sys.argv[i].startswith("--prompt-file="):
prompt_file = sys.argv[i].split("=", 1)[1]
i += 1
else:
i += 1
files = [f for f in files_from.read_text(encoding="utf-8").splitlines() if f.strip()]
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(
files, root, mode=cache_mode
files, root, mode=cache_mode, prompt_file=prompt_file
)
out = root / _GRAPHIFY_OUT
out.mkdir(parents=True, exist_ok=True)
Expand Down
4 changes: 4 additions & 0 deletions graphify/llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -1935,6 +1935,10 @@ def _checkpoint_chunk(result: dict, chunk: "list[Path | FileSlice]") -> None:
merge_existing=True,
allowed_source_files=allowed,
mode="deep" if deep_mode else None,
# Stamp the entry with the prompt that produced it, so a release
# that changes _EXTRACTION_SYSTEM re-extracts instead of replaying
# this vintage forever (#1939).
prompt=_extraction_system(deep=deep_mode),
)
except Exception as _exc: # noqa: BLE001 — checkpoint is best-effort
print(f"[graphify] incremental cache checkpoint failed: {_exc}", file=sys.stderr)
Expand Down
8 changes: 5 additions & 3 deletions graphify/skill-agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:

Before dispatching any subagents, check which files already have cached extraction results:

SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.

```bash
$(cat graphify-out/.graphify_python) -c "
import json
Expand All @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]

cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')

# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
Expand Down Expand Up @@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```

Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
Expand All @@ -311,7 +313,7 @@ from pathlib import Path

new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
Expand Down
8 changes: 5 additions & 3 deletions graphify/skill-amp.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:

Before dispatching any subagents, check which files already have cached extraction results:

SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.

```bash
$(cat graphify-out/.graphify_python) -c "
import json
Expand All @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]

cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')

# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
Expand Down Expand Up @@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```

Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
Expand All @@ -311,7 +313,7 @@ from pathlib import Path

new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
Expand Down
8 changes: 5 additions & 3 deletions graphify/skill-claw.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:

Before dispatching any subagents, check which files already have cached extraction results:

SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.

```bash
$(cat graphify-out/.graphify_python) -c "
import json
Expand All @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]

cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')

# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
Expand Down Expand Up @@ -305,7 +307,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```

Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
Expand All @@ -314,7 +316,7 @@ from pathlib import Path

new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
Expand Down
8 changes: 5 additions & 3 deletions graphify/skill-codex.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,8 @@ Before dispatching subagents, print a timing estimate:

Before dispatching any subagents, check which files already have cached extraction results:

SPEC_PATH below is the **absolute** path of the `references/extraction-spec.md` that ships beside this SKILL.md — the same file Step B2 loads and hands to every subagent. It is the extraction prompt, so cache entries are attributed to it: when a graphify upgrade changes the prompt, entries produced by the old one are re-extracted instead of replayed, and unchanged prompts keep their entries (#1939). Substitute the real path in both Step B0 and Step B3 — pass the same one to each, and do not drop the argument.

```bash
$(cat graphify-out/.graphify_python) -c "
import json
Expand All @@ -226,7 +228,7 @@ detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encodin
# every source file (#1392). Video is transcribed to a document in Step 2.5 first.
all_files = [f for cat in ('document', 'paper', 'image') for f in detect['files'].get(cat, [])]

cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH')
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files, root='INPUT_PATH', prompt_file='SPEC_PATH')

# Always (re)write the cache file: write hits, else DELETE any leftover from a prior
# run so Part C never merges a stale .graphify_cached.json (#1392).
Expand Down Expand Up @@ -302,7 +304,7 @@ print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens'
"
```

Save new results to cache:
Save new results to cache. Pass the same SPEC_PATH as Step B0 — it stamps each entry with the prompt that produced it, and a write under a different prompt than the read lands where the next run won't look (#1939):
```bash
$(cat graphify-out/.graphify_python) -c "
import json
Expand All @@ -311,7 +313,7 @@ from pathlib import Path

new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached, prompt_file='SPEC_PATH')
print(f'Cached {saved} files')
"
```
Expand Down
Loading