diff --git a/graphify/skill-agents.md b/graphify/skill-agents.md index 4969929c0..d0a541c61 100644 --- a/graphify/skill-agents.md +++ b/graphify/skill-agents.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-amp.md b/graphify/skill-amp.md index 4969929c0..d0a541c61 100644 --- a/graphify/skill-amp.md +++ b/graphify/skill-amp.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-claw.md b/graphify/skill-claw.md index bafbca074..c308242e1 100644 --- a/graphify/skill-claw.md +++ b/graphify/skill-claw.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-codex.md b/graphify/skill-codex.md index d71f4592d..e36e56150 100644 --- a/graphify/skill-codex.md +++ b/graphify/skill-codex.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message (Codex)** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the compact subagent prompt (rules, node Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-copilot.md b/graphify/skill-copilot.md index bafbca074..c308242e1 100644 --- a/graphify/skill-copilot.md +++ b/graphify/skill-copilot.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-devin.md b/graphify/skill-devin.md index c444a852d..e55ce01dc 100644 --- a/graphify/skill-devin.md +++ b/graphify/skill-devin.md @@ -228,7 +228,7 @@ else: Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -267,7 +267,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the write happens, so `.graphify_chunk_N.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch subagents** @@ -331,10 +333,10 @@ FILE_LIST Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_N.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only, or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. After each subagent call completes, write its result to `graphify-out/.graphify_chunk_N.json`. **After each subagent call completes, read the real token counts from the subagent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then merge: diff --git a/graphify/skill-droid.md b/graphify/skill-droid.md index ae1d49325..be8b795fe 100644 --- a/graphify/skill-droid.md +++ b/graphify/skill-droid.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-kilo.md b/graphify/skill-kilo.md index 6aa87a8bd..ba19292bf 100644 --- a/graphify/skill-kilo.md +++ b/graphify/skill-kilo.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-kiro.md b/graphify/skill-kiro.md index bafbca074..c308242e1 100644 --- a/graphify/skill-kiro.md +++ b/graphify/skill-kiro.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-opencode.md b/graphify/skill-opencode.md index 8c3ce3eed..6b7eef403 100644 --- a/graphify/skill-opencode.md +++ b/graphify/skill-opencode.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message (OpenCode)** @@ -268,10 +270,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-pi.md b/graphify/skill-pi.md index bafbca074..c308242e1 100644 --- a/graphify/skill-pi.md +++ b/graphify/skill-pi.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-trae.md b/graphify/skill-trae.md index af59166ae..ba73dd5af 100644 --- a/graphify/skill-trae.md +++ b/graphify/skill-trae.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -274,10 +276,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-vscode.md b/graphify/skill-vscode.md index 11a2837b5..17508ae1e 100644 --- a/graphify/skill-vscode.md +++ b/graphify/skill-vscode.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch subagents and paste their responses** @@ -272,10 +274,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill-windows.md b/graphify/skill-windows.md index 0dc73947a..f86f38aff 100644 --- a/graphify/skill-windows.md +++ b/graphify/skill-windows.md @@ -228,7 +228,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -265,7 +265,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -298,10 +300,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/graphify/skill.md b/graphify/skill.md index bafbca074..c308242e1 100644 --- a/graphify/skill.md +++ b/graphify/skill.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-agents.md b/tools/skillgen/expected/graphify__skill-agents.md index 4969929c0..d0a541c61 100644 --- a/tools/skillgen/expected/graphify__skill-agents.md +++ b/tools/skillgen/expected/graphify__skill-agents.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-amp.md b/tools/skillgen/expected/graphify__skill-amp.md index 4969929c0..d0a541c61 100644 --- a/tools/skillgen/expected/graphify__skill-amp.md +++ b/tools/skillgen/expected/graphify__skill-amp.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-claw.md b/tools/skillgen/expected/graphify__skill-claw.md index bafbca074..c308242e1 100644 --- a/tools/skillgen/expected/graphify__skill-claw.md +++ b/tools/skillgen/expected/graphify__skill-claw.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-codex.md b/tools/skillgen/expected/graphify__skill-codex.md index d71f4592d..e36e56150 100644 --- a/tools/skillgen/expected/graphify__skill-codex.md +++ b/tools/skillgen/expected/graphify__skill-codex.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message (Codex)** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the compact subagent prompt (rules, node Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-copilot.md b/tools/skillgen/expected/graphify__skill-copilot.md index bafbca074..c308242e1 100644 --- a/tools/skillgen/expected/graphify__skill-copilot.md +++ b/tools/skillgen/expected/graphify__skill-copilot.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-devin.md b/tools/skillgen/expected/graphify__skill-devin.md index c444a852d..e55ce01dc 100644 --- a/tools/skillgen/expected/graphify__skill-devin.md +++ b/tools/skillgen/expected/graphify__skill-devin.md @@ -228,7 +228,7 @@ else: Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -267,7 +267,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the write happens, so `.graphify_chunk_N.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch subagents** @@ -331,10 +333,10 @@ FILE_LIST Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_N.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only, or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. After each subagent call completes, write its result to `graphify-out/.graphify_chunk_N.json`. **After each subagent call completes, read the real token counts from the subagent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then merge: diff --git a/tools/skillgen/expected/graphify__skill-droid.md b/tools/skillgen/expected/graphify__skill-droid.md index ae1d49325..be8b795fe 100644 --- a/tools/skillgen/expected/graphify__skill-droid.md +++ b/tools/skillgen/expected/graphify__skill-droid.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-kilo.md b/tools/skillgen/expected/graphify__skill-kilo.md index 6aa87a8bd..ba19292bf 100644 --- a/tools/skillgen/expected/graphify__skill-kilo.md +++ b/tools/skillgen/expected/graphify__skill-kilo.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-kiro.md b/tools/skillgen/expected/graphify__skill-kiro.md index bafbca074..c308242e1 100644 --- a/tools/skillgen/expected/graphify__skill-kiro.md +++ b/tools/skillgen/expected/graphify__skill-kiro.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-opencode.md b/tools/skillgen/expected/graphify__skill-opencode.md index 8c3ce3eed..6b7eef403 100644 --- a/tools/skillgen/expected/graphify__skill-opencode.md +++ b/tools/skillgen/expected/graphify__skill-opencode.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message (OpenCode)** @@ -268,10 +270,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-pi.md b/tools/skillgen/expected/graphify__skill-pi.md index bafbca074..c308242e1 100644 --- a/tools/skillgen/expected/graphify__skill-pi.md +++ b/tools/skillgen/expected/graphify__skill-pi.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-trae.md b/tools/skillgen/expected/graphify__skill-trae.md index af59166ae..ba73dd5af 100644 --- a/tools/skillgen/expected/graphify__skill-trae.md +++ b/tools/skillgen/expected/graphify__skill-trae.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -274,10 +276,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-vscode.md b/tools/skillgen/expected/graphify__skill-vscode.md index 11a2837b5..17508ae1e 100644 --- a/tools/skillgen/expected/graphify__skill-vscode.md +++ b/tools/skillgen/expected/graphify__skill-vscode.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch subagents and paste their responses** @@ -272,10 +274,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill-windows.md b/tools/skillgen/expected/graphify__skill-windows.md index 0dc73947a..f86f38aff 100644 --- a/tools/skillgen/expected/graphify__skill-windows.md +++ b/tools/skillgen/expected/graphify__skill-windows.md @@ -228,7 +228,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -265,7 +265,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -298,10 +300,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/expected/graphify__skill.md b/tools/skillgen/expected/graphify__skill.md index bafbca074..c308242e1 100644 --- a/tools/skillgen/expected/graphify__skill.md +++ b/tools/skillgen/expected/graphify__skill.md @@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch ALL subagents in a single message** @@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/fragments/core/core.md b/tools/skillgen/fragments/core/core.md index a5e117b5c..05b818698 100644 --- a/tools/skillgen/fragments/core/core.md +++ b/tools/skillgen/fragments/core/core.md @@ -165,7 +165,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],' Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -202,7 +202,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. @@DISPATCH@@ @@ -211,10 +213,10 @@ Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-2 Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run: ```bash diff --git a/tools/skillgen/fragments/core/devin.md b/tools/skillgen/fragments/core/devin.md index c444a852d..e55ce01dc 100644 --- a/tools/skillgen/fragments/core/devin.md +++ b/tools/skillgen/fragments/core/devin.md @@ -228,7 +228,7 @@ else: Before dispatching subagents, print a timing estimate: - Load `total_words` and file counts from `graphify-out/.graphify_detect.json` -- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25) +- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1) - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit)) - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys" @@ -267,7 +267,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt **Step B1 - Split into chunks** -Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. +Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted. + +Each subagent must write its entire chunk's extraction in a single call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the write happens, so `.graphify_chunk_N.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything. **Step B2 - Dispatch subagents** @@ -331,10 +333,10 @@ FILE_LIST Wait for all subagents. For each result: - Check that `graphify-out/.graphify_chunk_N.json` exists on disk — this is the success signal - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache -- If the file is missing, the subagent was likely dispatched as read-only — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip. +- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only, or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip. - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort -If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used. +If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`. After each subagent call completes, write its result to `graphify-out/.graphify_chunk_N.json`. **After each subagent call completes, read the real token counts from the subagent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then merge: