Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ A long or bundled instruction has enough surface area for a subtle misread: the
### #3 — Sizing the run becomes guesswork
Some work is run with too much horsepower because "stronger" feels safer; some is run too cheaply until the failure costs more than the saved tokens. The same guessing hits the reasoning-effort dial: max it on everything and burn tokens, or skim a task that needed real deliberation. All of it is guessing wearing operational clothing.

**The fix → `modelchk`:** from one risk read, recommend two coordinates: the cheapest sufficient capability tier (`fast`, `standard`, `frontier`) and the reasoning effort within it (`glance` to `exhaustive`), each on a neutral scale. It advises; it never routes, pins, or names a concrete model or level.
**The fix → `modelchk`:** from one risk read, recommend two coordinates: the cheapest sufficient capability tier (`fast`, `standard`, `frontier`) and the reasoning effort within it (`glance` to `exhaustive`), each on a neutral scale. It also names the task domain (`coding`, `agentic`, `reasoning`, `writing`, `knowledge`, `general`) the tier is measured in, since models of one price class differ by domain. It advises; it never routes, pins, or names a concrete model or level.

<details>
<summary><b>[PROOF]</b></summary>
Expand Down
31 changes: 24 additions & 7 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ A long or bundled instruction has enough surface area for a subtle misread: the
### #3 — Sizing the run becomes guesswork
Some work is run with too much horsepower because "stronger" feels safer; some is run too cheaply until the failure costs more than the saved tokens. The same guessing hits the reasoning-effort dial: max it on everything and burn tokens, or skim a task that needed real deliberation. All of it is guessing wearing operational clothing.

**The fix → `modelchk`:** from one risk read, recommend two coordinates: the cheapest sufficient capability tier (`fast`, `standard`, `frontier`) and the reasoning effort within it (`glance` to `exhaustive`), each on a neutral scale. It advises; it never routes, pins, or names a concrete model or level.
**The fix → `modelchk`:** from one risk read, recommend two coordinates: the cheapest sufficient capability tier (`fast`, `standard`, `frontier`) and the reasoning effort within it (`glance` to `exhaustive`), each on a neutral scale. It also names the task domain (`coding`, `agentic`, `reasoning`, `writing`, `knowledge`, `general`) the tier is measured in, since models of one price class differ by domain. It advises; it never routes, pins, or names a concrete model or level.

<details>
<summary><b>[PROOF]</b></summary>
Expand Down Expand Up @@ -509,6 +509,21 @@ Size the run before spending it: how strong a model, and how hard it should thin

The axes are independent. A bounded but fiddly task can be `fast` + `thorough`; a quick expert call can be `frontier` + `glance`. In most work they move together, parting when a cheap task needs hard thinking or a strong model needs only a quick call. Effort buys deliberation, never capability, and more effort is not more correct.

**Task domain** is the kind of capability the tier is measured in. It sizes nothing. It tells whoever binds the tier which measured strength matters, because models of one price class differ by domain:

- `coding`: writing, changing, or reviewing code in a repository.
- `agentic`: long multi-step tool use, where the work is completing a run of actions without derailing.
- `reasoning`: math, logic, planning, or analysis whose answer is derived rather than looked up.
- `writing`: prose meant for a reader, such as documents, messages, or copy.
- `knowledge`: research and factual recall across sources.
- `general`: none of the above dominates.

A tier is a capability claim, never a price claim. A binder resolves it against measured capability in the named domain, read live from a regularly refreshed evaluation source the operator has enabled, and uses price only to choose the cheapest model among those that measure sufficient. A measured model with no known price stays eligible and ranks after the priced ones. Equal measurements give an equal tier whatever order the candidates are listed in, and one model offered through two routes is one model. Among sufficient models at one price, the higher measurement wins before listing order. Price rank stands in for capability only when no source is enabled, the source cannot be read, or it measures fewer than two of the candidates, and the binder says which. Three limits hold on any source:

- An explicit user mapping of a model to a tier outranks every measurement and every price rank.
- A model the source does not list is unmeasured, not weak: it stays out of the measured ranking until a user mapping places it, and the binder says so.
- The source's terms decide what may be shown. Open data can be reported with its scores and attribution; data read under the user's own credential informs the choice and stays internal, so only the attribution and the resulting tier are shown.

## Workflow

1. Frame the exact work unit: task, artifact, review, rerun, or plan.
Expand All @@ -521,19 +536,20 @@ The axes are independent. A bounded but fiddly task can be `fast` + `thorough`;
- cost of a wrong answer.
3. Choose the cheapest capability tier whose ceiling covers the work's judgment and risk.
4. Default effort to track tier (`fast`→`glance`, `standard`→`measured`, `frontier`→`thorough`), reserving `exhaustive` for the hardest, highest-stakes work. Raise effort when an otherwise cheap task needs ambiguity resolved, long multi-step reasoning, or adversarial self-check; lower it for bounded work under a strong model.
5. Report both coordinates, one shared rationale, `move up if...` and `move down if...` triggers for each dial, and the proof surface: verification still needed regardless of tier or effort.
6. Stop.
5. Name the one task domain the work depends on most. Pick `general` rather than guess when none dominates.
6. Report both coordinates and the domain, one shared rationale, `move up if...` and `move down if...` triggers for each dial, and the proof surface: verification still needed regardless of tier or effort.
7. Stop.

## Rules

- Size two dials only: capability tier and reasoning effort. Call count and skill order belong to routing, owned by the workflow recommendation.
- Size two dials only: capability tier and reasoning effort. The task domain labels which capability the tier is measured in and sizes nothing. Call count and skill order belong to routing, owned by the workflow recommendation.
- Size cognitive spend only: which capability tier and how much reasoning effort. Context budget, fan-out width, and tool-permission scope are orchestration dials and stay outside this skill, even when they share the risk assessment.
- Use neutral language: `fast`/`standard`/`frontier` for tier and `glance`/`measured`/`thorough`/`exhaustive` for effort. Effort names positions on the active model's ladder (floor, default, above-default, ceiling), never vendor levels.
- Use neutral language: `fast`/`standard`/`frontier` for tier, `glance`/`measured`/`thorough`/`exhaustive` for effort, and one of the six words above for domain. Effort names positions on the active model's ladder (floor, default, above-default, ceiling), never vendor levels.
- Bind effort to positions, not level names. If a model lacks an interior level, collapse that rung to the nearest available setting; never resolve outside its range.
- Default to the cheapest sufficient tier and the effort the work needs, not the strongest of either.
- Effort budgets deliberation, not capability or answer length. Never raise effort to buy capability; raise the tier.
- This skill has no routing authority. The user, harness, or executor decides what runs and sets the actual level.
- Do not name concrete model products, vendors, or versions in durable mechanism text.
- Do not name concrete model products, vendors, versions, or evaluation sources in durable mechanism text.
- Risk beats size: a one-file high-risk change can need `frontier`; a broad mechanical rename can stay `fast` + `thorough` with complete verification.
- Verification is separate. Neither dial replaces tests, review, command output, manual QA, or other proof.
- Do not execute the task being sized, change configuration, call another model, or alter provider settings.
Expand All @@ -544,6 +560,7 @@ The axes are independent. A bounded but fiddly task can be `fast` + `thorough`;
```text
recommended_tier: fast|standard|frontier
recommended_effort: glance|measured|thorough|exhaustive
task_domain: coding|agentic|reasoning|writing|knowledge|general
rationale: <one sentence, covering both dials>
move_up_if: <signals that would justify a stronger tier or higher effort>
move_down_if: <signals that would justify a cheaper tier or lower effort>
Expand All @@ -552,7 +569,7 @@ proof_surface: <verification still required, independent of tier and effort>

Before finishing, confirm the report:

- names exactly one tier (`fast`/`standard`/`frontier`) and one effort (`glance`/`measured`/`thorough`/`exhaustive`);
- names exactly one tier (`fast`/`standard`/`frontier`), one effort (`glance`/`measured`/`thorough`/`exhaustive`), and one task domain;
- states the cheapest sufficient tier and the effort the work needs, not the strongest of either;
- gives move-up and move-down triggers covering both dials;
- names the proof surface;
Expand Down
29 changes: 23 additions & 6 deletions skills/depth/modelchk/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,21 @@ Size the run before spending it: how strong a model, and how hard it should thin

The axes are independent. A bounded but fiddly task can be `fast` + `thorough`; a quick expert call can be `frontier` + `glance`. In most work they move together, parting when a cheap task needs hard thinking or a strong model needs only a quick call. Effort buys deliberation, never capability, and more effort is not more correct.

**Task domain** is the kind of capability the tier is measured in. It sizes nothing. It tells whoever binds the tier which measured strength matters, because models of one price class differ by domain:

- `coding`: writing, changing, or reviewing code in a repository.
- `agentic`: long multi-step tool use, where the work is completing a run of actions without derailing.
- `reasoning`: math, logic, planning, or analysis whose answer is derived rather than looked up.
- `writing`: prose meant for a reader, such as documents, messages, or copy.
- `knowledge`: research and factual recall across sources.
- `general`: none of the above dominates.

A tier is a capability claim, never a price claim. A binder resolves it against measured capability in the named domain, read live from a regularly refreshed evaluation source the operator has enabled, and uses price only to choose the cheapest model among those that measure sufficient. A measured model with no known price stays eligible and ranks after the priced ones. Equal measurements give an equal tier whatever order the candidates are listed in, and one model offered through two routes is one model. Among sufficient models at one price, the higher measurement wins before listing order. Price rank stands in for capability only when no source is enabled, the source cannot be read, or it measures fewer than two of the candidates, and the binder says which. Three limits hold on any source:

- An explicit user mapping of a model to a tier outranks every measurement and every price rank.
- A model the source does not list is unmeasured, not weak: it stays out of the measured ranking until a user mapping places it, and the binder says so.
- The source's terms decide what may be shown. Open data can be reported with its scores and attribution; data read under the user's own credential informs the choice and stays internal, so only the attribution and the resulting tier are shown.

## Workflow

1. Frame the exact work unit: task, artifact, review, rerun, or plan.
Expand All @@ -36,19 +51,20 @@ The axes are independent. A bounded but fiddly task can be `fast` + `thorough`;
- cost of a wrong answer.
3. Choose the cheapest capability tier whose ceiling covers the work's judgment and risk.
4. Default effort to track tier (`fast`→`glance`, `standard`→`measured`, `frontier`→`thorough`), reserving `exhaustive` for the hardest, highest-stakes work. Raise effort when an otherwise cheap task needs ambiguity resolved, long multi-step reasoning, or adversarial self-check; lower it for bounded work under a strong model.
5. Report both coordinates, one shared rationale, `move up if...` and `move down if...` triggers for each dial, and the proof surface: verification still needed regardless of tier or effort.
6. Stop.
5. Name the one task domain the work depends on most. Pick `general` rather than guess when none dominates.
6. Report both coordinates and the domain, one shared rationale, `move up if...` and `move down if...` triggers for each dial, and the proof surface: verification still needed regardless of tier or effort.
7. Stop.

## Rules

- Size two dials only: capability tier and reasoning effort. Call count and skill order belong to routing, owned by the workflow recommendation.
- Size two dials only: capability tier and reasoning effort. The task domain labels which capability the tier is measured in and sizes nothing. Call count and skill order belong to routing, owned by the workflow recommendation.
- Size cognitive spend only: which capability tier and how much reasoning effort. Context budget, fan-out width, and tool-permission scope are orchestration dials and stay outside this skill, even when they share the risk assessment.
- Use neutral language: `fast`/`standard`/`frontier` for tier and `glance`/`measured`/`thorough`/`exhaustive` for effort. Effort names positions on the active model's ladder (floor, default, above-default, ceiling), never vendor levels.
- Use neutral language: `fast`/`standard`/`frontier` for tier, `glance`/`measured`/`thorough`/`exhaustive` for effort, and one of the six words above for domain. Effort names positions on the active model's ladder (floor, default, above-default, ceiling), never vendor levels.
- Bind effort to positions, not level names. If a model lacks an interior level, collapse that rung to the nearest available setting; never resolve outside its range.
- Default to the cheapest sufficient tier and the effort the work needs, not the strongest of either.
- Effort budgets deliberation, not capability or answer length. Never raise effort to buy capability; raise the tier.
- This skill has no routing authority. The user, harness, or executor decides what runs and sets the actual level.
- Do not name concrete model products, vendors, or versions in durable mechanism text.
- Do not name concrete model products, vendors, versions, or evaluation sources in durable mechanism text.
- Risk beats size: a one-file high-risk change can need `frontier`; a broad mechanical rename can stay `fast` + `thorough` with complete verification.
- Verification is separate. Neither dial replaces tests, review, command output, manual QA, or other proof.
- Do not execute the task being sized, change configuration, call another model, or alter provider settings.
Expand All @@ -59,6 +75,7 @@ The axes are independent. A bounded but fiddly task can be `fast` + `thorough`;
```text
recommended_tier: fast|standard|frontier
recommended_effort: glance|measured|thorough|exhaustive
task_domain: coding|agentic|reasoning|writing|knowledge|general
rationale: <one sentence, covering both dials>
move_up_if: <signals that would justify a stronger tier or higher effort>
move_down_if: <signals that would justify a cheaper tier or lower effort>
Expand All @@ -67,7 +84,7 @@ proof_surface: <verification still required, independent of tier and effort>

Before finishing, confirm the report:

- names exactly one tier (`fast`/`standard`/`frontier`) and one effort (`glance`/`measured`/`thorough`/`exhaustive`);
- names exactly one tier (`fast`/`standard`/`frontier`), one effort (`glance`/`measured`/`thorough`/`exhaustive`), and one task domain;
- states the cheapest sufficient tier and the effort the work needs, not the strongest of either;
- gives move-up and move-down triggers covering both dials;
- names the proof surface;
Expand Down
Loading