fix(skill): size extraction chunks by output, not just file count

graphify/skill-agents.md

@@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],'
 
 Before dispatching subagents, print a timing estimate:
 - Load `total_words` and file counts from `graphify-out/.graphify_detect.json`
-- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25)
+- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1)
 - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit))
 - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys"
 
@@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt
 
 **Step B1 - Split into chunks**
 
-Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted.
+Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted.
+
+Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything.
 
 **Step B2 - Dispatch ALL subagents in a single message**
 
@@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema,
 Wait for all subagents. For each result:
 - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal
 - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache
-- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip.
+- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip.
 - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort
 
-If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used.
+If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`.
 
 Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run:
 ```bash

graphify/skill-amp.md

@@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],'
 
 Before dispatching subagents, print a timing estimate:
 - Load `total_words` and file counts from `graphify-out/.graphify_detect.json`
-- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25)
+- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1)
 - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit))
 - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys"
 
@@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt
 
 **Step B1 - Split into chunks**
 
-Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted.
+Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted.
+
+Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything.
 
 **Step B2 - Dispatch ALL subagents in a single message**
 
@@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema,
 Wait for all subagents. For each result:
 - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal
 - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache
-- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip.
+- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip.
 - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort
 
-If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used.
+If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`.
 
 Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run:
 ```bash

graphify/skill-claw.md

@@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],'
 
 Before dispatching subagents, print a timing estimate:
 - Load `total_words` and file counts from `graphify-out/.graphify_detect.json`
-- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25)
+- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1)
 - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit))
 - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys"
 
@@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt
 
 **Step B1 - Split into chunks**
 
-Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted.
+Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted.
+
+Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything.
 
 **Step B2 - Dispatch ALL subagents in a single message**
 
@@ -276,10 +278,10 @@ See `references/extraction-spec.md` for the exact subagent prompt (JSON schema,
 Wait for all subagents. For each result:
 - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal
 - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache
-- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip.
+- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip.
 - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort
 
-If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used.
+If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`.
 
 Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run:
 ```bash

graphify/skill-codex.md

@@ -206,7 +206,7 @@ Path('graphify-out/.graphify_semantic.json').write_text(json.dumps({'nodes':[],'
 
 Before dispatching subagents, print a timing estimate:
 - Load `total_words` and file counts from `graphify-out/.graphify_detect.json`
-- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25)
+- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25), or `ceil(uncached_non_code_files / 12)` if `DEEP_MODE=true` (chunk size is 10-15 — see Step B1)
 - Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit))
 - Print: "Semantic extraction: ~N files → X agents, estimated ~Ys"
 
@@ -243,7 +243,9 @@ Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt
 
 **Step B1 - Split into chunks**
 
-Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted.
+Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each (10-15 when `DEEP_MODE=true` — deep mode extracts materially more INFERRED edges per file, so the same file count produces much more output). Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted.
+
+Each subagent must write its entire chunk's extraction in a single `Write` call within a single turn, and a turn's output is capped (~64k output tokens). A chunk whose expected output is too large for one turn doesn't get truncated — the subagent's turn simply ends before the `Write` call happens, so `.graphify_chunk_NN.json` never appears at all (see #1758). If a corpus is unusually dense (long files, heavy cross-referencing) even at the reduced deep-mode chunk size, tell the subagent explicitly to prioritize the most cross-referenced entities and keep extraction proportionate (e.g. "~60 nodes / ~80 edges for this chunk") rather than exhaustively extracting everything.
 
 **Step B2 - Dispatch ALL subagents in a single message (Codex)**
 
@@ -273,10 +275,10 @@ See `references/extraction-spec.md` for the compact subagent prompt (rules, node
 Wait for all subagents. For each result:
 - Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal
 - If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache
-- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip.
+- If the file is missing, there are two possible causes and you cannot tell them apart from the missing file alone — print a warning covering both: "chunk N missing from disk — either the subagent was dispatched as read-only (Explore type instead of general-purpose), or its output exceeded the model's per-turn token limit while generating the chunk (more likely for large/deep-mode chunks — see #1758). Re-run with a smaller chunk size and/or `subagent_type="general-purpose"`." Do not silently skip.
 - If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort
 
-If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used.
+If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used, and to reduce chunk size (e.g. halve it) if the missing chunks were large or `DEEP_MODE=true`.
 
 Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run:
 ```bash