Skip to content

fix(detect): let .graphifyinclude rescue generic keyword skips (#1225)#1921

Open
HerenderKumar wants to merge 1 commit into
Graphify-Labs:v8from
HerenderKumar:include-overrides-sensitive-skip
Open

fix(detect): let .graphifyinclude rescue generic keyword skips (#1225)#1921
HerenderKumar wants to merge 1 commit into
Graphify-Labs:v8from
HerenderKumar:include-overrides-sensitive-skip

Conversation

@HerenderKumar

Copy link
Copy Markdown

Fixes #1225.

Problem

The generic secrets keyword heuristic drops any file whose stem ends in token(s)/secret/credential/… — swallowing benign docs like references/prompts/derive-locked-tokens.md — and nothing can override it. The drop is also invisible: neither the CLI output nor GRAPH_REPORT.md mentions skipped files at all on current v8.

Root cause of the missing escape hatch: the .graphifyinclude machinery (_load_graphifyinclude, _is_included) is still loaded in detect() but never consulted — orphaned when dot-dirs became indexed by default — so the allowlist literally cannot rescue anything, matching its own docstring's disclaimer.

Fix

Suggested fixes 2 + 3 from the issue:

  • An explicit .graphifyinclude entry beats the keyword heuristic. _is_sensitive is split into a reason-returning core (_sensitive_reason) so the scan can tell the heuristic stage apart from the specific ones; the bool wrapper keeps the existing API and tests binding. Only wildcard-free entries count as explicit intent, and only the keyword stage can be overridden — secrets dirs, .env, key files, id_rsa, … are never rescued, so a broad * glob written to opt hidden files into traversal cannot silently start ingesting api_token.txt.
  • The drop is no longer silent. Keyword skips are annotated in skipped_sensitive with the escape hatch spelled out, one aggregated stderr note per scan names the affected files (capped at 6, like the Dockerfile/Gemfile/Makefile/Rakefile/LICENSE and other extensionless, non-shebang project files currently produce no visible trace in graphify's output — not classified, not counted anywhere #1692 unclassified note), and GRAPH_REPORT.md gains a ## Skipped Files section listing every skipped file by name so a swallowed file is greppable. detect_incremental delegates to detect(), so incremental scans inherit all of this.

The _credentials.py source-file case from the follow-up comment is already handled on v8 by the #1666 source-code exemption; this covers the remaining docs/data case.

Tests

Six new tests, written first and watched fail: exact-path rescue, annotation + stderr note, glob entries don't rescue, credential stores are never rescued, report lists names, report omits the section when empty. Full detect/report/hypergraph suites pass (186/186); the repo-wide run matches the pre-change baseline exactly (this env is missing tree-sitter/watch/dedup extras, identical failures with and without the change).

…ify-Labs#1225)

The secrets keyword heuristic (stem ends in token/secret/credential/...)
silently dropped benign files like derive-locked-tokens.md, and the
.graphifyinclude allowlist could not override it: the include machinery
was loaded in detect() but never consulted after dot-dirs became indexed
by default, so the allowlist literally could not rescue anything.

Wire the allowlist into the sensitive check as an explicit escape hatch:

- A wildcard-free .graphifyinclude entry now rescues a file from the
  keyword stage. Glob entries deliberately do not count, and the
  specific patterns (secrets dirs, .env, key files) are never
  overridden, so a broad * cannot start ingesting api_token.txt.
- Keyword skips are annotated in skipped_sensitive and surfaced once
  per scan on stderr, naming the escape hatch.
- GRAPH_REPORT.md lists every skipped file by name (previously the
  report did not mention skips at all), so a swallowed file is
  greppable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Secrets heuristic silently drops benign *-tokens.md docs; .graphifyinclude cannot override the sensitive skip

1 participant