Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions packages/skill-catalog/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- `import-bundle --clean`: replace the catalog content for the bundle's
scope instead of merging. The pre-clean state (store copies +
registry entries) is snapshotted to
`.devin-ecosystem/skill-catalog/import-snapshots/` first, each store
copy is staged and swapped atomically (`os.rename`), and a mid-run
failure rolls the touched items back — fail-closed, never a
half-replaced catalog. `import-bundle --restore SNAPSHOT_DIR`
recovers a snapshot manually (plan-first like every mutation);
snapshots are kept until pruned. Items still land `quarantined`:
`approved`/`active` items are demoted, already-quarantined items get
a same-state `Registry.replace` history entry, `retired` items are
skipped, and items absent from the bundle are untouched.
- `lint --slop`: a deterministic content pass over each item's markdown
(SKILL.md, rule files, bundled `.md`s) with five regex rules —
`slop-fingerprint` (stock LLM phrases), `slop-emdash` (dash clause
separators; ranges like `3.10–3.14` exempt), `slop-sycophancy`
(assistant openers/closers), `slop-not-just-but` and `slop-filler`.
Fenced code blocks are skipped; `slop-allow` / `slop-allow:<rule-id>`
line markers and `slop-allow-file` suppress findings so legitimate
content is never blocked by a bare denylist. Findings carry rule id +
line refs and exit 1; `--warn-only` reports WARN and exits 0.
- `g3 ITEM --judge --evidence PATH [--out PATH]` — produce a
`g3-report/0.1` document by asking poordjaevin (the optional `judge`
extra) three concrete yes/no questions about the promotion evidence.
Expand Down
52 changes: 48 additions & 4 deletions packages/skill-catalog/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,9 @@ content.
`.devin/rules/*.md`, hashes each item (sha256), annotates it with its
registry state.
- **Lint** — structural checks: frontmatter presence, `name` matching
the directory, a real `description`, `# ` titles on rules.
the directory, a real `description`, `# ` titles on rules. `lint
--slop` adds a deterministic content pass: five regex heuristics for
LLM-flavored prose, each with a per-line/per-file allowlist escape.
- **Gates** — two offline evidence gates (G1/G2, below) plus a G3
promotion gate driven by `g3-report` verdicts. G1/G2 are heuristics,
**not proof**.
Expand Down Expand Up @@ -83,6 +85,7 @@ PyPI install produce the same CLI.
```bash
devin-skill-catalog scan [PATH ...] # inventory (read-only)
devin-skill-catalog lint [PATH ...] # structural findings
[--slop] [--warn-only] # deterministic anti-slop rules
devin-skill-catalog diff A B # inventory diff by sha256
devin-skill-catalog gate g1 PATH # static hygiene gate
devin-skill-catalog gate g2 PATH [--packs-dir D] # grounding gate
Expand All @@ -96,6 +99,8 @@ devin-skill-catalog activate ITEM # approved → active
devin-skill-catalog retire ITEM # any state → retired
devin-skill-catalog export-bundle --out DIR # pack approved items
devin-skill-catalog import-bundle PATH # land items quarantined
[--clean] # replace item copies the bundle carries
[--restore SNAP] # recover a pre-clean snapshot
```

`ITEM` is `kind:name` (`skill:foo`, `rule:bar`) or a bare name when it
Expand Down Expand Up @@ -124,6 +129,44 @@ The only write targets are the registry file and quarantine store under
the Devin config dir, plus whatever you point `--out` at. Quarantine
snapshots the item; removing the original from `.devin/` is your call.

**Clean imports.** `import-bundle` merges by default: an item that is
already registered keeps its stored copy and the import skips it.
`import-bundle --clean` *replaces* the catalog content for the bundle's
scope instead — the stored copy of every item the bundle carries is
swapped for the bundle's version. Safely, in order:

1. the pre-clean state (store copies + registry entries) is snapshotted
to `.devin-ecosystem/skill-catalog/import-snapshots/<ts>/` first;
2. each new copy is staged in a sibling dir and swapped in with
`os.rename` — no half-written stores;
3. a mid-run failure rolls the touched items back, and
`import-bundle --restore <snapshot> --apply` recovers the same state
manually. Snapshots are kept until you prune them.

Clean still lands everything `quarantined` (`approved`/`active` items
are demoted; `retired` items are skipped — a clean import does not
resurrect a deliberate end-state), and items absent from the bundle are
untouched.

**Slop lint.** `lint --slop PATH` runs the deterministic anti-slop
rules over each item's markdown (SKILL.md, rule files, bundled `.md`s):

| rule | what it flags |
|------|---------------|
| `slop-fingerprint` | stock LLM phrases — `delve into`, `seamless`, `robust`, `comprehensive`, `it's worth noting`, `in summary`/`in conclusion`, sentence-initial `Additionally,`/`Moreover,`/`Furthermore,` |
| `slop-emdash` | `—` (and spaced `–`) used as a clause separator — not en-dashes in ranges like `3.10–3.14` |
| `slop-sycophancy` | assistant-style openers/closers — `Great question`, `Happy to help`, `Hope this helps`, `Let me know`, `You're absolutely right` |
| `slop-not-just-but` | the `not just X but Y` escalation (also `not only/merely/simply`) |
| `slop-filler` | hype/throat-clearing — `in today's fast-paced world`, `cutting-edge`, `game-changer`, `a testament to`, `plays a vital role`, `at the end of the day`, `first and foremost`, `when it comes to`, `in the world of` |

Lines inside fenced code blocks are skipped (examples are not prose),
and every rule honors the same allowlist escapes — a finding is
suppressed when its line carries `slop-allow`, `slop-allow:<rule-id>`,
or the file carries `slop-allow-file` (put the marker in an HTML
comment: `<!-- slop-allow:slop-emdash -->`). Findings print with the
rule id and line number and exit 1; `--warn-only` reports them as WARN
and exits 0.

## Lifecycle

```
Expand Down Expand Up @@ -271,9 +314,10 @@ Rules of the road:
quarantine on the receiving side.
- On upstream changes, re-vendor by **replacing** the staged item dir
(delete it, then copy) and re-quarantining from scratch — copying over
the old tree leaves files upstream deleted, and the quarantine store
merges rather than replaces, so stale files would survive the
re-review.
the old tree leaves files upstream deleted, and the plain
`import-bundle` merges rather than replaces, so stale files would
survive the re-review (`import-bundle --clean` is the replacing
variant).

Worked example: 25 skills from `addyosmani/agent-skills` (MIT), 6 from
`DietrichGebert/ponytail` (MIT) and a local `graphify` install were
Expand Down
19 changes: 14 additions & 5 deletions packages/skill-catalog/STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@
block scalars), degrades gracefully.
- `lint.py` — structural findings: frontmatter block, `name` vs dir,
real `description`, `# ` title on rules.
- `slop.py` — deterministic anti-slop rules (`slop-fingerprint`,
`slop-emdash`, `slop-sycophancy`, `slop-not-just-but`, `slop-filler`)
over item markdown; fenced code skipped; `slop-allow[:<rule-id>]` /
`slop-allow-file` allowlist markers.
- `gates.py` — G1 (lint + injection/exfil/`curl|sh`/destructive-shell
phrasing + secret-shaped strings, values suppressed) and G2 (declared
files/commands grounded, body references WARN, `--packs-dir` verifies
Expand All @@ -36,23 +40,28 @@
- `bundle.py` — export `approved` items to dir/`.tar[.gz]` with
`manifest.json` (items, per-file sha256, source profile); import
verifies checksums and lands items `quarantined`, never active.
- `cli.py` — `scan`, `lint`, `diff`, `gate g1|g2`, `g3 --judge`,
`--clean` replaces the store copies the bundle carries (pre-clean
snapshot to `import-snapshots/`, atomic `os.rename` swap, rollback on
failure); `--restore` recovers a snapshot.
- `cli.py` — `scan`, `lint` (structural + `--slop` anti-slop mode),
`diff`, `gate g1|g2`, `g3 --judge`,
`quarantine`,
`promote` (runs G1 on the stored copy, `--force` overrides G1;
enforces the G3 policy — always-on rules require a `g3-report` with
`improves`/`no-detectable-effect`, `regresses` hard-blocks every kind
even under `--force`, `inconclusive` skills need
`--g3-inconclusive-reason`), `activate`, `retire`, `export-bundle`,
`import-bundle`. Mutations are plan-first: no writes without
`import-bundle` (`--clean` safe replacement, `--restore` recovery).
Mutations are plan-first: no writes without
`--apply`.
- Tests: 123 (+1 when the `judge` extra is installed), all on
- Tests: 216 (+1 when the `judge` extra is installed), all on
synthetic `.devin/` fixture trees; judge tests use scripted
clients/backends — never a real model.

## Verified

- `python -m pytest packages/skill-catalog` — 124 passed (Linux /
Python 3.14, poordjaevin installed), 123 + 1 skip without it.
- `python -m pytest packages/skill-catalog` — 217 passed (Linux /
Python 3.14, poordjaevin installed), 216 + 1 skip without it.
- Manual end-to-end: scan → quarantine --apply → promote --apply
(G1 pass) → export-bundle → import-bundle --apply lands quarantined.
- Manual `g3 --judge --evidence` run through a real poordjaevin
Expand Down
3 changes: 2 additions & 1 deletion packages/skill-catalog/llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,11 @@
## Key commands
- `devin-skill-catalog scan [PATH ...]`: inventory skills/rules (read-only)
- `devin-skill-catalog lint|gate g1|gate g2 PATH`: findings, exit 1 on FAIL
- `devin-skill-catalog lint --slop PATH`: deterministic anti-slop rules (`slop-fingerprint|emdash|sycophancy|not-just-but|filler`; `slop-allow[:<id>]`/`slop-allow-file` markers suppress; `--warn-only` exits 0)
- `devin-skill-catalog g3 ITEM --judge --evidence FILE [--out R.json]`: produce a g3-report via poordjaevin (optional `judge` extra; `POORDJAEVIN_BACKEND` selects the backend, default keyless local NLI)
- `devin-skill-catalog quarantine|promote|activate|retire ITEM --apply`: lifecycle mutations (plan printed without `--apply`)
- `devin-skill-catalog diff A B`: inventory diff by content hash
- `devin-skill-catalog export-bundle --out DIR` / `import-bundle PATH --apply`: move approved items, import lands quarantined
- `devin-skill-catalog export-bundle --out DIR` / `import-bundle PATH --apply`: move approved items, import lands quarantined (`--clean` replaces the copies the bundle carries — pre-clean snapshot + atomic swap + rollback; `--restore SNAP` recovers)
- `pipx install "devin-skill-catalog @ git+https://github.com/Icaro0310/devin-devkit.git#subdirectory=packages/skill-catalog"`: install the CLI
- `python -m pytest`: run the test suite (synthetic fixtures only)

Expand Down
Loading
Loading