Skip to content

fix(cli): redact secret-shaped values from the model bundle - #39

Merged
jameslcowan merged 2 commits into
mainfrom
fix/redact-model-bundle
Oct 26, 2026
Merged

jameslcowan merged 2 commits into
mainfrom
fix/redact-model-bundle

Conversation

@jameslcowan

@jameslcowan jameslcowan commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

What changed

Problem: buildBundle in cli/src/generate.js copied up to 10 hotspot files into the prompt bundle as they were, about 60,000 characters. A key hard-coded in a hotspot file went to the model provider, to the agent CLI that --via names, or into .complex-md/prompt.md for the user's agent.

Change:

  • New cli/src/redact.js, with no dependency. It replaces each secret-shaped value with [redacted:<rule>]. 11 rules:
    • aws-access-key: AKIA, ASIA, ABIA, ACCA and A3T prefixes, 16 characters of [A-Z2-7].
    • github-token: ghp_, gho_, ghu_, ghs_, ghr_ with 36 characters, and github_pat_ with 22 and 59.
    • anthropic-key: sk-ant-<kind><2 digits>- and 32 or more characters.
    • openai-key: the T3BlbkFJ marker, and sk-proj-, sk-svcacct-, sk-admin- with 40 or more characters.
    • slack-token (xoxb- and the other xox kinds, xapp-) and slack-webhook (hooks.slack.com).
    • stripe-key: sk_live_ and rk_live_.
    • google-api-key: AIza and 35 characters.
    • private-key: a BEGIN ... PRIVATE KEY block to its END line, or to the end of the base64 when the END line is missing.
    • jwt: 3 base64url parts, the first 2 starting with eyJ.
    • generic-assignment (heuristic): a value assigned with =, :=, : or => to a name that holds key, secret, token, password, pwd or credential. The value is replaced only when its Shannon entropy is 3.5 bits per character or more and it holds a digit and a letter. An environment lookup, a template slot, a placeholder word, a URL, and a lowercase lookup key of short parts (such as web-article-v1) stay. Only the value is replaced, so the name and the quotes stay.
      The token formats and expressions are written for this project.
  • excerpt() redacts each file whole before it cuts the excerpt, so a cut never leaves half of a key that no rule matches. A second pass covers the rest of the bundle (paths, blind spots).
  • buildBundle returns redacted (counts per rule). generate prints 1 line when the count is above 0, for example redacted 3 secret-shaped values in the prompt bundle (aws-access-key 1, generic-assignment 2); each is sent as [redacted:<rule>]. The line never holds a value.
  • The front matter that generate writes is computed from the signals and never passes through redaction. Template prose does not read the excerpts, so the template path writes the same map.
  • Public texts that describe what the model call carries now say so: --help, cli/README.md (Privacy), SECURITY.md and the FAQ in content/index.html. The README also says that a secret in another shape is sent as it is.

Co-change partners: cli/bin/complex-md.js changed with cli/src/generate.js (the help text). cli/README.md changed, and content/spec.md did not: its 2 sentences on the model call (the "One tool, two entry points" list and the FAQ) stay true, and a change to the spec text needs its own source.

Limits: redaction matches shapes, so a secret in a format with no rule, or a short or low-entropy password, is still sent. A credential file by path is already dropped from the map by cli/src/secrets.js.

How it was tested

  • cd cli && npm test: exit 0, 806 tests, 804 pass, 0 fail, 2 todo (the 2 todo tests were there before).
  • cd cli && node --test test/truth.test.js test/site.test.js test/home.test.js: 60 of 60 pass.
  • npm run build at the root: passes.
  • New cli/test/redact.test.js (10 tests):
    • each rule replaces its fake token, and the fixtures cover every rule;
    • the assignment rule keeps the name and the quotes, and takes bare values;
    • a private key without its END line is replaced;
    • 18 near misses and lines that only name a secret stay byte for byte;
    • every file in cli/src and cli/bin, and the generation prompt, pass through with 0 replacements;
    • the log line names rules and counts, never a value;
    • buildBundle on a repository whose top hotspot holds every fake value: no original value in the bundle, every rule placeholder present, the computed front matter verbatim;
    • a bundle with nothing to redact is unchanged;
    • generate --template writes the same map as templateMap, and the saved .complex-md/prompt.md holds no original value.
  • cli/test/generate.test.js: a stub provider records the request. The request holds no original value, the run prints 1 count line, and the written map has the computed front matter.
  • cli/test/generate-via.test.js: the stub claude CLI now saves its stdin. The stdin holds no original value.
  • The fake tokens in cli/test/fixtures/fake-secrets.mjs are plainly fake (FAKE, fake, 0123) and built from fragments at run time, so no whole token is in the repository.
  • A check outside the suite: on 1 private repository of 434 JavaScript and config files, every remaining match was a fake key in a test fixture. The scan took 0.3 s.

AI assistance

Claude (Claude Code) wrote the code, the tests, the text changes and this description.

  • An AI tool helped write this change. The tool and the part it wrote are stated above.
  • Each commit the tool helped write has an Assisted-by: or a Co-Authored-By: trailer.
  • A person read the whole diff and will answer the review comments.

🤖 Generated with Claude Code

Problem: buildBundle copied up to 10 hotspot files into the prompt
bundle as they were, about 60,000 characters. A key hard-coded in a
hotspot file went to the model provider, to the --via agent CLI, or
into .complex-md/prompt.md for the user's agent.

Add cli/src/redact.js, with no dependency: 10 rules for known token
formats (AWS, GitHub, OpenAI, Anthropic, Slack, Stripe, Google, private
key blocks, JSON web tokens) and 1 assignment rule that keeps a value
only when its entropy is 3.5 bits per character or more and it holds a
digit. Each match becomes [redacted:<rule>]. excerpt() redacts each file
whole before the cut, and a second pass covers the rest of the bundle.
generate prints 1 line with the count per rule, never a value. The
front matter that generate writes never passes through redaction.

Tests: 10 in redact.test.js (each rule, near misses, the CLI's own
source unchanged, the template path), 1 in generate.test.js on the
provider request, 1 in generate-via.test.js on the stub CLI's stdin.
npm test: 806 tests, 0 failures.

Intent: keep hard-coded credentials in hotspot files out of every
third party that reads the bundle.
Assisted-by: Claude
Problem: --help, cli/README.md, SECURITY.md and the site FAQ said the
model call carries the hotspot excerpts, with no word on the values
that the previous commit now replaces.

Each text now says that each secret-shaped value goes as
[redacted:<rule>]. The README names the rule families and says that a
secret in another shape is sent as it is. truth, site and home tests:
60 of 60 pass. npm run build at the root passes.

Intent: describe exactly what the model call carries.
Assisted-by: Claude
@github-actions

Copy link
Copy Markdown

COMPLEX.md: 1 hotspot touched, 3 co-change partners not in this pull request.

Hotspot Score Churn Fixes Fan in Tests
cli/src/generate.js 2780 13 7 5 10
  • cli/src/generate.js: Before editing this file, open parseFrontMatter in cli/src/complexmd.js, because it reads back the exact lines that frontMatter writes.

Co-change partners not in this pull request

  • cli/README.md changed. Unchanged: content/spec.md (8 commits together, 53 percent). Say whether each needs a change.
  • content/index.html changed. Unchanged: content/spec.md (6 commits together, 40 percent), site/site.css (5 commits together, 45 percent). Say whether each needs a change.

Read from COMPLEX.md at base c3c51d5. The map was generated 2026-10-25 at da42d08, 3 commits behind the base. complex-md-action runs no code from this pull request.

@jameslcowan
jameslcowan merged commit bf61aa6 into main Oct 26, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant