Skip to content

feat(route): legacy ladder and per-linter custom-rule counts #577

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
theCodeDrift merged 3 commits into from
Oct 23, 2026
Merged

feat(route): legacy ladder and per-linter custom-rule counts #577

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ce52bc4
feat(detect): count the repository's own rules per linter
theCodeDrift Oct 23, 2026
e02b94a
feat(route): apply the legacy ladder and describe each detected linter
theCodeDrift Oct 23, 2026
ed49196
fix(detect): skip an unreadable ESLint rule directory instead of fail…
theCodeDrift Oct 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/legacy-linter-recipes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"@taskless/cli": patch
---

Rules for a linter the repository already runs now follow a fixed order. The agent first looks for a rule the linter already ships, checking each tool's catch-all rules by name (`no-restricted-syntax`, `banned-api`, `noRestrictedImports`, and others). It writes a new custom rule in that linter only when the repository already has five or more of its own to model it on. Otherwise the request becomes a Taskless rule. `route` (topic v8) no longer asks the user to choose between a linter rule and a Taskless rule, and it describes each linter `detect` found in the repository. During onboarding (topic v11), the agent only enables rules the repository's installed packages already ship.

`detect --json` reports `customRules` for ESLint, Biome, pylint, flake8, and Ruff: how many rules the repository wrote itself for that linter, and where they are. A count that cannot be read without running a JavaScript config is reported as `"unknown"`.
2 changes: 2 additions & 0 deletions openspec/changes/legacy-linter-recipes/.openspec.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-10-22
78 changes: 78 additions & 0 deletions openspec/changes/legacy-linter-recipes/design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
## Context

`route` already names `create-legacy-rule` for "a tool the repo already runs", onboarding lists `legacy` as a candidate destination, and the report has a `legacy` section that counts toward its three-rule gate. The gap is the recipe, which is generic and teaches nothing tool-specific. See proposal.md for motivation.

Constraints that shape the approach:

- Recipes are embedded at build time (`src/prompts/recipes.ts`). Variants already exist as `<topic>.anonymous.md`, selected by `--anonymous`.
- `route` already has runtime-rendered slots (`%(DETECT_EVIDENCE)s`, `%(HOST_TOOLS)s`), so a per-linter paragraph is the same mechanism with a new slot.
- `detectRuleStyles` (`src/detect/scan.ts`) only recognizes ESLint local rule directories and plugin packages, and reports styles, not counts.
- Unlike Vale and ast-grep, Taskless does not pin these linters. The installed version is whatever the repository has, so recipe claims must hold across the versions a repository is likely to run, or say where they diverge.

## Goals / Non-Goals

**Goals:**

- An agent that reaches a legacy destination finds and enables an existing rule correctly, including the traps for its tool and version.
- The decision to write a new custom rule rests on a count the CLI computes, not on the agent's impression.

**Non-Goals:**

- Teaching rule authoring for any linter. Step 2 learns from the repository's examples only.
- Running external linters from `taskless check`.
- Recipes for linters beyond the five launch tools. The generic recipe still covers them.
- Matching a request to a named rule inside the CLI. That stays the agent's job, so `detect`'s "no packaged-rule catalog matching" requirement is unchanged.

## Decisions

**Per-tool variants reuse the `.anonymous` mechanism, keyed by `detect`'s linter name.** `create-legacy-rule.<tool>.md` files are collected by the same glob. `--tool` takes precedence over `--anonymous`, since no legacy recipe has an anonymous variant. Keying on `detect`'s `name` means the agent copies a value from JSON into a flag with no interpretation. Keying on language was rejected: Python alone maps to Ruff, pylint, and flake8, which have different dialects. `--tool=anonymous` is refused so it cannot collide with the anonymous variant file.

**`route` paragraphs render at serve time from a scan.** `route` is served with a fresh `detect` scan, the same way `HOST_TOOLS` is, and gains one paragraph per detected linter that has a legacy recipe. A static table of every tool was rejected: it grows with every linter we add, and an agent in a Ruff repository gains nothing from reading about Biome. With `mechanics === false` (recipes served without a CLI), the slot renders the full table, because there is no repository to scan.

**The five-rule threshold is a `detect` count, not recipe prose.** The threshold decides whether an agent may write code into a linter, so it should be computed rather than judged. `detect` already reads the config files involved. Counting rules means:

- **ESLint:** rule files and rule folders in a house rule directory (`eslint-rules`, `eslint-local-rules`, `tools/eslint-rules`). Index files, tests, type declarations, and helper folders are not counted. A flat config that imports a plugin from a relative path or defines rules inline, and a local-rules loader package, are custom rules whose count cannot be read without executing JavaScript, so they are reported as evidence with an `"unknown"` count. A counted directory of five or more settles it either way. Otherwise unknown stays unknown, and the recipe treats it as below the threshold.
- **Biome:** the length of `plugins` across the root and nested configs.
- **pylint:** `load-plugins` entries that resolve to a module inside the repository.
- **flake8:** checkers under `[flake8:local-plugins]`.

**Discovery for Ruff and Biome is web-first, then confirmed locally.** Both tools ship too many rules to match a request against: `ruff rule --all --output-format json` returned 972 rules on 0.17.2. External testing found no reliable way for an agent to search that list for a request. The web finds the candidate. `ruff rule <name>` and `biome explain <rule>` confirm that it exists in the installed version and reveal preview status.

**The ESLint plugin list is curated and linked, and the recipe never lists individual rules.** Eight plugins, each with a summary and a docs link. This narrows the standing "no catalog" line instead of removing it, because a catalog of rules goes stale with every plugin release, while a list of plugins changes rarely.

**Vendor tests pin recipe claims against real installs.** Each claim a recipe makes about tool behavior gets a test that runs the tool, in the same spirit as the existing Vale and ast-grep vendor-contract tests: preview status in the `ruff rule` JSON, `--help-msg` failing without `--load-plugins`, `biome explain` output, and `banned-modules` in `flake8-tidy-imports --help`. They run through `uvx` and `pnpm dlx` at pinned versions, so a version bump that breaks a claim fails a test instead of shipping a wrong recipe.

## Research record

Measured on 2026-10-22 with Ruff 0.17.2, pylint 4.1.3, flake8 7.4.1, Biome 2.6.2, and ESLint 10.13.0 current on npm. Doc quotes were fetched from official pages the same day.

| Tool | Claim | Source |
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------- |
| Ruff | Default set grew to 413 rules in 0.16.0, so `select` cannot be compared against `E4/E7/E9/F` | https://github.com/astral-sh/ruff/blob/main/changelogs/0.16.x.md |
| Ruff | `banned-api` is TID251, configured under `flake8-tidy-imports` | https://docs.astral.sh/ruff/settings/ and `ruff rule TID251` |
| Ruff | `unnecessary-regular-expression-compile` is preview-only and needs `--preview` | `ruff rule unnecessary-regular-expression-compile` |
| Ruff | No custom-rule API exists | no official page describes one |
| ESLint | v10 removed `.eslintrc*`; config lookup starts from each linted file | https://eslint.org/docs/latest/use/migrate-to-10.0.0 |
| ESLint | Local and inline plugins are supported in flat config | https://eslint.org/docs/latest/use/configure/plugins |
| ESLint | `eslint-plugin-import` 2.32.0, `eslint-plugin-react` 7.37.5, `eslint-plugin-jsx-a11y` 6.10.2 declare `eslint` peers only up to 9. `eslint-plugin-import-x` 4.17.1 declares `^10.0.0` | `npm view <pkg> peerDependencies.eslint` |
| Biome | GritQL plugins since 2.0; JS plugins a developer preview since 2.6 | https://biomejs.dev/linter/plugins/, https://biomejs.dev/blog/biome-v2-6/ |
| Biome | `biome explain <rule>` prints a rule's documentation | `biome explain noRestrictedImports` on 2.6.2 |
| pylint | `--list-msgs` lists 393 messages; `--help-msg=bad-builtin` fails without `--load-plugins=pylint.extensions.bad_builtin` | measured |
| pylint | `load-plugins`, `deprecated-modules`, `preferred-modules` | https://pylint.readthedocs.io/en/stable/user_guide/configuration/all-options.html |
| flake8 | Reads only `setup.cfg`, `tox.ini`, `.flake8`; `[flake8:local-plugins]` since 3.5.0 | https://flake8.pycqa.org/en/latest/user/configuration.html |
| flake8 | No rule-listing command; `--bug-report` lists installed plugins | measured |
| flake8 | `flake8-tidy-imports` provides `--banned-modules` | `flake8 --help` with the plugin installed |

Two claims are inferred, not quoted, and are not used in recipe text until verified: that flake8 has no native `pyproject.toml` support, and that `--rulesdir` is dead in ESLint 10.

## Risks / Trade-offs

- **A recipe claim drifts when a linter releases.** → The vendor tests pin each claim at a version. Bumping that version is a deliberate step that reruns them.
- **The repository runs an older linter than the recipe was measured against.** → Recipes state version boundaries where they matter (ESLint 9 against 10, Biome 1 against 2, Ruff before and after 0.16), and confirm discovery against the installed version.
- **The ESLint count is undercounted when config is computed.** → Unknown counts are treated as below the threshold, which sends the request to Taskless, the safe direction.
- **Web discovery is unavailable on the host.** → The recipe falls back to the local confirm command plus the catch-all rules, and says the search was skipped.
- **Recommending an `eslint-plugin-import` to an ESLint 10 repository installs with a peer warning.** → The recipe names `import-x` for that case and prefers whatever is already installed.

## Migration Plan

Recipe and additive JSON changes only. The `detect --json` schema gains a field, which existing readers ignore. Rollback is a revert of the recipe files. There is no state to migrate.
53 changes: 53 additions & 0 deletions openspec/changes/legacy-linter-recipes/proposal.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
## Why

When a repository already runs a linter, `create-legacy-rule` hands the agent 64 generic lines that say "mine the repo, then search the web". Each agent works out every linter's dialect from scratch, so it misses rules the tool already ships, gets traps like preview-only rules wrong, and writes rules into tools the team does not maintain. We are adding linters, and teaching an agent to author rules for hundreds of them is not sustainable. What is sustainable is teaching it to use what each tool already has.

## What Changes

- **A legacy ladder replaces the open-ended legacy destination.** For a request in a repository with a detected linter, `route` applies these steps in order:
1. **Always: would a rule the tool already ships solve this?** Each tool has a step it always takes. Ruff always tries `banned-api` (TID251). ESLint always tries a built-in rule, then a curated plugin list. Biome always checks whether an existing rule makes the problem go away.
2. **Only when the repository already has five or more of its own custom rules for that linter:** author another one, using those rules as the reference examples, not the tool's documentation.
3. A Taskless `sg` or `vale` rule.
4. A Taskless runtime rule.
- **Taskless does not teach linter rule authoring.** The legacy recipes carry no instructions for writing a custom ESLint rule, a pylint checker, a flake8 plugin, a Biome GritQL or JS plugin, or anything else. Step 2 learns only from the repository's own examples. A check that needs custom code and has no such examples belongs to Taskless.
- **Onboarding skips step 2 and never adds a dependency.** Onboarding runs logged out and writes no executable code, so it uses only step 1, and only with rules from packages the repository already has installed. A plugin that is not installed is something to recommend outside onboarding, after the user agrees to add it.
- **`taskless agent create-legacy-rule --tool=<name>`** returns a per-tool recipe. `<name>` is the linter `name` that `detect --json` reports. An unknown tool falls back to the generic recipe. Tools that cannot hold a rule of this kind (`ast-grep` and `vale`, which have their own recipes, and the formatters `prettier` and `black`) are refused with a pointer to the right recipe.
- **Per-tool recipes at launch: Ruff, ESLint (including typescript-eslint), Biome, pylint, flake8.** TSLint is not supported. Each recipe covers how to find an existing rule for a request, the tool's catch-all rules, version-specific traps, and how to run the tool so the user can see the rule fire:
- **Ruff and Biome:** find a rule by web search of the rules page, then confirm it against the installed version with `ruff rule <name>` or `biome explain <rule>`. Both tools have too many rules to match a request against locally. Ruff marks preview-only rules, and enabling one without `preview = true` means it never fires.
- **ESLint:** core rules first, with `no-restricted-syntax` and `no-restricted-imports` (and their `@typescript-eslint/` versions) as the catch-alls. Then a curated list of eight plugins, each with a one-line summary and a docs link.
- **pylint:** `--list-msgs` and `--help-msg`, with `deprecated-modules`, `preferred-modules`, and `bad-functions` as the catch-alls.
- **flake8:** no discovery step. The catch-all is `banned-modules` from `flake8-tidy-imports`.
- **`detect` counts the repository's own rules per linter.** `detect --json` adds a count of custom rules for each linter, which is the signal step 2 reads. The count covers local ESLint plugins and rule directories, Biome `plugins` entries, pylint `load-plugins` modules inside the repository, and flake8 `[flake8:local-plugins]`. Built-in extensions and third-party packages do not count. Neither does a config file on its own.
- **`route` gives a paragraph to each linter it detects.** The CLI renders each paragraph at runtime, and only for linters `detect` found. Each one says what the tool is for, what it always tries first, and the `--tool` command to fetch.
- **The catalog prohibition is lifted, narrowly.** The standing requirement says `create-legacy-rule` SHALL NOT embed a Taskless-maintained catalog of linter rules. The curated ESLint plugin list and each tool's named catch-all rules are a catalog, so that line changes to allow a short, curated, linked list of rule sources. It still forbids a catalog of individual rules.
- **`route` stops asking the user when both legacy and a Taskless rule fit.** The ladder's order decides instead. This also resolves a conflict with onboarding, which told the agent not to ask.
- Topic versions bump: `route` v7 → v8, `create-legacy-rule` v2 → v3, `onboard` v10 → v11.

## Capabilities

### New Capabilities

- `cli-legacy-rule-recipes`: the `--tool` per-tool variants, the five launch recipes and what each must cover, the generic fallback, refused tools, and the rule that legacy recipes do not teach rule authoring.

### Modified Capabilities

- `cli-rule-routing`: "Existing recipe authors in the detected linter's dialect" is restated for the ladder and the narrowed catalog line. "Route asks the user when multiple paths fit" no longer poses legacy against static as a choice. A new requirement covers the ladder and the per-linter paragraphs.
- `cli-detect`: a new requirement covers the per-linter custom-rule count. "No packaged-rule catalog matching" is unchanged, because the curated list lives in recipes and is not part of detection.
- `cli-agent`: "Agent subcommand serves recipes for commands" gains the `--tool` flag and its resolution order.
- `cli-onboard`: a new requirement limits onboarding to step 1, with installed packages only.

## Delivery Shape

**Stacked, merging forward.** Each PR is safe in production on its own:

1. **The `detect` custom-rule count, the ladder in `route`, the per-linter paragraphs, and the generic `create-legacy-rule` v3 and onboard v11 that follow the ladder.** The paragraphs name only `create-legacy-rule` here, so nothing references a variant that does not exist yet. The changeset lands on this PR.
2. **`--tool`, the five per-tool recipes and their vendor tests.** The `route` paragraphs start naming `--tool=<name>`. This PR archives the change.

The package is pre-1.0 and every surface here is additive or recipe text, so the bump is `patch`.

## Impact

- **Code**: `packages/cli/src/detect/scan.ts` and the detect schema; `packages/cli/src/commands/agent.ts` (the `--tool` flag); `packages/cli/src/prompts/recipes.ts` (variant lookup, runtime-rendered linter paragraphs); `packages/cli/src/agent/` (`route.md`, `create-legacy-rule.md`, `onboard.md`, five new `create-legacy-rule.<tool>.md` files).
- **Tests**: vendor tests that pin each recipe's claims against a real install of that linter (preview status in `ruff rule --output-format json`, `--help-msg` needing `--load-plugins`, `biome explain` output). These need Ruff, Biome, pylint, flake8, and ESLint available in CI.
- **No new runtime dependency** for the CLI. Recipes recommend plugins; the CLI installs nothing.
- **Research**: the per-tool signals and traps come from official docs and measured runs, recorded in `design.md` with the source for each.
Loading
Loading