From bb2b6e0d61800acf72afeef9f78058e190811bc3 Mon Sep 17 00:00:00 2001 From: STiFLeR7 Date: Thu, 16 Jul 2026 10:46:01 +0530 Subject: [PATCH] fix(skill): size extraction chunks by output, not just file count Part B2 dispatches Claude/Devin Agent-tool subagents to extract a chunk's semantic graph and write it in one Write call within a single turn. Step B1 sized chunks purely by input file count (20-25), with no accounting for expected output volume -- which is much higher with --mode deep (more INFERRED edges per file). A chunk whose output exceeds the ~64k-output-token turn limit doesn't get truncated: the subagent's turn simply ends before the Write call happens, so the chunk file never appears on disk at all, and that chunk's extraction is silently lost entirely. Step B3's existing diagnosis for a missing chunk file only considered one cause (Explore-type read-only dispatch), which is indistinguishable from an over-the-limit crash from the orchestrator's side. Changes (core.md and devin.md fragments -- the two platforms that dispatch subagents this way; aider.md's Step B2 is sequential single-context processing with no per-turn Write-call ceiling, so it isn't affected and wasn't touched): - Reduce chunk size to 10-15 files when DEEP_MODE=true. - Note the per-turn output ceiling explicitly, with guidance to keep extraction proportionate for unusually dense corpora. - Give Step B3 a second diagnosis branch for missing chunk files. - Update the pre-dispatch timing/agent-count estimate to account for the smaller deep-mode chunk size. Regenerated all derived skill.md/skill-.md artifacts and expected/ snapshots via `python -m tools.skillgen --bless`; `python -m tools.skillgen --check` passes clean. Only the platforms whose fragment source changed (core.md, devin.md) have real content diffs -- verified via `git diff --stat` after blessing, not just `git status` (which flags many more files due to autocrlf line-ending bookkeeping with no actual content change). Fixes #1758 --- graphify/skill-agents.md | 10 ++++++---- graphify/skill-amp.md | 10 ++++++---- graphify/skill-claw.md | 10 ++++++---- graphify/skill-codex.md | 10 ++++++---- graphify/skill-copilot.md | 10 ++++++---- graphify/skill-devin.md | 10 ++++++---- graphify/skill-droid.md | 10 ++++++---- graphify/skill-kilo.md | 10 ++++++---- graphify/skill-kiro.md | 10 ++++++---- graphify/skill-opencode.md | 10 ++++++---- graphify/skill-pi.md | 10 ++++++---- graphify/skill-trae.md | 10 ++++++---- graphify/skill-vscode.md | 10 ++++++---- graphify/skill-windows.md | 10 ++++++---- graphify/skill.md | 10 ++++++---- tools/skillgen/expected/graphify__skill-agents.md | 10 ++++++---- tools/skillgen/expected/graphify__skill-amp.md | 10 ++++++---- tools/skillgen/expected/graphify__skill-claw.md | 10 ++++++---- tools/skillgen/expected/graphify__skill-codex.md | 10 ++++++---- tools/skillgen/expected/graphify__skill-copilot.md | 10 ++++++---- tools/skillgen/expected/graphify__skill-devin.md | 10 ++++++---- tools/skillgen/expected/graphify__skill-droid.md | 10 ++++++---- tools/skillgen/expected/graphify__skill-kilo.md | 10 ++++++---- tools/skillgen/expected/graphify__skill-kiro.md | 10 ++++++---- tools/skillgen/expected/graphify__skill-opencode.md | 10 ++++++---- tools/skillgen/expected/graphify__skill-pi.md | 10 ++++++---- tools/skillgen/expected/graphify__skill-trae.md | 10 ++++++---- tools/skillgen/expected/graphify__skill-vscode.md | 10 ++++++---- tools/skillgen/expected/graphify__skill-windows.md | 10 ++++++---- tools/skillgen/expected/graphify__skill.md | 10 ++++++---- tools/skillgen/fragments/core/core.md | 10 ++++++---- tools/skillgen/fragments/core/devin.md | 10 ++++++---- 32 files changed, 192 insertions(+), 128 deletions(-) diff --git a/graphify/skill-agents.md b/graphify/skill-agents.md index 4969929c0..d0a541c61 100644 --- a/graphify/skill-agents.md +++ b/graphify/skill-agents.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-amp.md b/graphify/skill-amp.md index 4969929c0..d0a541c61 100644 --- a/graphify/skill-amp.md +++ b/graphify/skill-amp.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-claw.md b/graphify/skill-claw.md index bafbca074..c308242e1 100644 --- a/graphify/skill-claw.md +++ b/graphify/skill-claw.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-codex.md b/graphify/skill-codex.md index d71f4592d..e36e56150 100644 --- a/graphify/skill-codex.md +++ b/graphify/skill-codex.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message (Codex)** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the compact subagent prompt (rules, node Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-copilot.md b/graphify/skill-copilot.md index bafbca074..c308242e1 100644 --- a/graphify/skill-copilot.md +++ b/graphify/skill-copilot.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-devin.md b/graphify/skill-devin.md index c444a852d..e55ce01dc 100644 --- a/graphify/skill-devin.md +++ b/graphify/skill-devin.md @@ -228,7 +228,7 @@ else: Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -267,7 +267,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the write happens, so `.graphify_chunk_N.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch subagents** @@ -331,10 +333,10 @@ FILE_LIST Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_N.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only, or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. After each subagent call completes, write its result to `graphify-out/.graphify_chunk_N.json`. **After each subagent call completes, read the real token counts from the subagent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then merge: diff --git a/graphify/skill-droid.md b/graphify/skill-droid.md index ae1d49325..be8b795fe 100644 --- a/graphify/skill-droid.md +++ b/graphify/skill-droid.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-kilo.md b/graphify/skill-kilo.md index 6aa87a8bd..ba19292bf 100644 --- a/graphify/skill-kilo.md +++ b/graphify/skill-kilo.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-kiro.md b/graphify/skill-kiro.md index bafbca074..c308242e1 100644 --- a/graphify/skill-kiro.md +++ b/graphify/skill-kiro.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-opencode.md b/graphify/skill-opencode.md index 8c3ce3eed..6b7eef403 100644 --- a/graphify/skill-opencode.md +++ b/graphify/skill-opencode.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message (OpenCode)** @@ -268,10 +270,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-pi.md b/graphify/skill-pi.md index bafbca074..c308242e1 100644 --- a/graphify/skill-pi.md +++ b/graphify/skill-pi.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-trae.md b/graphify/skill-trae.md index af59166ae..ba73dd5af 100644 --- a/graphify/skill-trae.md +++ b/graphify/skill-trae.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -274,10 +276,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-vscode.md b/graphify/skill-vscode.md index 11a2837b5..17508ae1e 100644 --- a/graphify/skill-vscode.md +++ b/graphify/skill-vscode.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch subagents and paste their responses** @@ -272,10 +274,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-windows.md b/graphify/skill-windows.md index 0dc73947a..f86f38aff 100644 --- a/graphify/skill-windows.md +++ b/graphify/skill-windows.md @@ -228,7 +228,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -265,7 +265,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -298,10 +300,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill.md b/graphify/skill.md index bafbca074..c308242e1 100644 --- a/graphify/skill.md +++ b/graphify/skill.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-agents.md b/tools/skillgen/expected/graphify__skill-agents.md index 4969929c0..d0a541c61 100644 --- a/tools/skillgen/expected/graphify__skill-agents.md +++ b/tools/skillgen/expected/graphify__skill-agents.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-amp.md b/tools/skillgen/expected/graphify__skill-amp.md index 4969929c0..d0a541c61 100644 --- a/tools/skillgen/expected/graphify__skill-amp.md +++ b/tools/skillgen/expected/graphify__skill-amp.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-claw.md b/tools/skillgen/expected/graphify__skill-claw.md index bafbca074..c308242e1 100644 --- a/tools/skillgen/expected/graphify__skill-claw.md +++ b/tools/skillgen/expected/graphify__skill-claw.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-codex.md b/tools/skillgen/expected/graphify__skill-codex.md index d71f4592d..e36e56150 100644 --- a/tools/skillgen/expected/graphify__skill-codex.md +++ b/tools/skillgen/expected/graphify__skill-codex.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message (Codex)** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the compact subagent prompt (rules, node Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-copilot.md b/tools/skillgen/expected/graphify__skill-copilot.md index bafbca074..c308242e1 100644 --- a/tools/skillgen/expected/graphify__skill-copilot.md +++ b/tools/skillgen/expected/graphify__skill-copilot.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-devin.md b/tools/skillgen/expected/graphify__skill-devin.md index c444a852d..e55ce01dc 100644 --- a/tools/skillgen/expected/graphify__skill-devin.md +++ b/tools/skillgen/expected/graphify__skill-devin.md @@ -228,7 +228,7 @@ else: Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -267,7 +267,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the write happens, so `.graphify_chunk_N.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch subagents** @@ -331,10 +333,10 @@ FILE_LIST Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_N.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only, or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. After each subagent call completes, write its result to `graphify-out/.graphify_chunk_N.json`. **After each subagent call completes, read the real token counts from the subagent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then merge: diff --git a/tools/skillgen/expected/graphify__skill-droid.md b/tools/skillgen/expected/graphify__skill-droid.md index ae1d49325..be8b795fe 100644 --- a/tools/skillgen/expected/graphify__skill-droid.md +++ b/tools/skillgen/expected/graphify__skill-droid.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-kilo.md b/tools/skillgen/expected/graphify__skill-kilo.md index 6aa87a8bd..ba19292bf 100644 --- a/tools/skillgen/expected/graphify__skill-kilo.md +++ b/tools/skillgen/expected/graphify__skill-kilo.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-kiro.md b/tools/skillgen/expected/graphify__skill-kiro.md index bafbca074..c308242e1 100644 --- a/tools/skillgen/expected/graphify__skill-kiro.md +++ b/tools/skillgen/expected/graphify__skill-kiro.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-opencode.md b/tools/skillgen/expected/graphify__skill-opencode.md index 8c3ce3eed..6b7eef403 100644 --- a/tools/skillgen/expected/graphify__skill-opencode.md +++ b/tools/skillgen/expected/graphify__skill-opencode.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message (OpenCode)** @@ -268,10 +270,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-pi.md b/tools/skillgen/expected/graphify__skill-pi.md index bafbca074..c308242e1 100644 --- a/tools/skillgen/expected/graphify__skill-pi.md +++ b/tools/skillgen/expected/graphify__skill-pi.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-trae.md b/tools/skillgen/expected/graphify__skill-trae.md index af59166ae..ba73dd5af 100644 --- a/tools/skillgen/expected/graphify__skill-trae.md +++ b/tools/skillgen/expected/graphify__skill-trae.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -274,10 +276,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-vscode.md b/tools/skillgen/expected/graphify__skill-vscode.md index 11a2837b5..17508ae1e 100644 --- a/tools/skillgen/expected/graphify__skill-vscode.md +++ b/tools/skillgen/expected/graphify__skill-vscode.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch subagents and paste their responses** @@ -272,10 +274,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-windows.md b/tools/skillgen/expected/graphify__skill-windows.md index 0dc73947a..f86f38aff 100644 --- a/tools/skillgen/expected/graphify__skill-windows.md +++ b/tools/skillgen/expected/graphify__skill-windows.md @@ -228,7 +228,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -265,7 +265,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -298,10 +300,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill.md b/tools/skillgen/expected/graphify__skill.md index bafbca074..c308242e1 100644 --- a/tools/skillgen/expected/graphify__skill.md +++ b/tools/skillgen/expected/graphify__skill.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/fragments/core/core.md b/tools/skillgen/fragments/core/core.md index a5e117b5c..05b818698 100644 --- a/tools/skillgen/fragments/core/core.md +++ b/tools/skillgen/fragments/core/core.md @@ -165,7 +165,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -202,7 +202,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. @@DISPATCH@@ @@ -211,10 +213,10 @@ Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-2 Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/fragments/core/devin.md b/tools/skillgen/fragments/core/devin.md index c444a852d..e55ce01dc 100644 --- a/tools/skillgen/fragments/core/devin.md +++ b/tools/skillgen/fragments/core/devin.md @@ -228,7 +228,7 @@ else: Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -267,7 +267,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the write happens, so `.graphify_chunk_N.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch subagents** @@ -331,10 +333,10 @@ FILE_LIST Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_N.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only, or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. After each subagent call completes, write its result to `graphify-out/.graphify_chunk_N.json`. **After each subagent call completes, read the real token counts from the subagent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then merge: