Secrets heuristic silently drops benign *-tokens.md docs; .graphifyinclude cannot override the sensitive skip

[dawtips]

Summary

detect._is_sensitive → _generic_keyword_hit silently drops any file whose stem ends in token/tokens ("keyword ends the stem → names the contents"). That rule is right for api_token.txt / github-personal-access-token, but it also swallows ordinary docs whose topic is tokens — and there is no escape hatch: .graphifyinclude's docstring says sensitive files "are still excluded later", and no config/env overrides the skip.

Observed (graphifyy 0.8.36)

A prompt-template file in a resume-tooling repo:

references/prompts/derive-locked-tokens.md

It's an LLM prompt about locking proper-noun tokens during a rewrite — pure markdown, no credentials. detect() returned it in skipped_sensitive, so the file was missing from the graph entirely. Silent: the report shows only a count, so a single dropped file in a 200-file corpus is easy to miss until a query comes back empty.

Note the existing long-slug carve-out (token-economics-of-recall.md from #436/#718) doesn't help here, because the keyword is at the END of the stem — derive-locked-tokens is a 3-word descriptive slug that happens to end in the head noun.

Why renaming isn't a great answer

We did rename it (derive-token-locks.md) and that works, but it's the tool reshaping the project: the file was named after the function it feeds (derive_locked_tokens()), and any repo in the NLP/LLM space ("tokens" is core vocabulary) will keep tripping this.

Suggested fixes (any of)

1. Tighten the end-of-stem rule: require a credential-store extension (.txt, .env, .key, .json, extensionless) or a short (≤2-word) stem before treating an end-of-stem token(s) as a secret. A 3+-word kebab slug ending in tokens with a .md extension is a topic, not a credential store — same insight as the existing mid-phrase carve-out.
2. Let .graphifyinclude override the sensitive skip for explicitly allowlisted paths (opt-in, user-stated intent).
3. At minimum, list the skipped filenames in GRAPH_REPORT.md (not just a count) so the drop isn't silent.

[AstroMined]

Adding a data point: this also drops source code files, not just docs.

Observed on graphifyy 0.8.38, Python SDK repo:

src/generic_sdk/_credentials.py

Ordinary Python module (credential check/create operations for cluster nodes, no stored secrets). The stem _credentials trips both branches of _generic_keyword_hit: the keyword ends the stem, and the stem is a single word. detect() returns it in skipped_sensitive, so the module's classes and functions are absent from the graph entirely. On 0.8.5 the file was included; the drop appeared after upgrading to 0.8.37/0.8.38.

Also confirmed empirically on 0.8.38 that a .graphifyinclude containing the exact relative path does not rescue it, matching the docstring ("Sensitive files ... are still excluded later").

Worth noting this is the failure class from #718, which was closed as fixed in v0.7.6. The fix that shipped (the load-bearing/end-of-stem refinement) handles mid-phrase topic words, but the source-code-extension exemption proposed there was not part of it, so any code file named after the credential domain it manages (_credentials.py, password_reset.ts, token_validator.java) still vanishes silently.

Of the suggested fixes above, 2 (let .graphifyinclude override the skip, explicit user intent) plus the #718 extension exemption would cover both the docs case and the source-file case; 3 (list skipped filenames in GRAPH_REPORT.md) would at least make the drop visible. Happy to provide more repro detail if useful.

[HerenderKumar]

Taking a crack at this. Root cause on current v8: _load_graphifyinclude/_is_included still exist in detect.py, but nothing consults them anymore (orphaned when dot-dirs became indexed by default), so the allowlist can't rescue anything — matching the docstring you quoted.

Plan: an explicit .graphifyinclude match overrides the generic keyword heuristic (suggested fix 2), while the specific patterns (.env, key files, secrets dirs) stay hard-excluded so a broad include glob can't pull in real credential stores; skipped filenames get listed in GRAPH_REPORT.md instead of vanishing (fix 3), plus a scan-time note naming the escape hatch.

FWIW the source-file case from the follow-up comment (_credentials.py) is already handled on v8 — #1666 added a source-code exemption to the keyword stage. PR shortly.

[HerenderKumar]

PR up: #1921.