Skip to content

Add pinned real-world static canaries and benchmark regression tests - #62

Open
TristanKruse wants to merge 1 commit into
fix/pre-v1-auditfrom
test/real-world-canary
Open

TristanKruse wants to merge 1 commit into
fix/pre-v1-auditfrom
test/real-world-canary

Conversation

@TristanKruse

@TristanKruse TristanKruse commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Add static-only canaries against pinned Rack, Rake, and RSpec Core production lib sources. Never install corpus dependencies or execute its source, Gemfiles, gemspecs, or tests.
  • Verify both dependency-analysis modes using reviewed file/edge counts and deterministic internal-graph SHA-256 baselines. Check a passing public rule, a deliberately forbidden known import, assertion reporting, and warm-cache reuse.
  • Record cold/warm stage timings and available heap/RSS metrics as JSON artifacts. Add a read-only, relevant-PR/manual workflow with three isolated jobs and broad runaway limits. No recurring schedule yet; keep advisory until stability/cost are reviewed.
  • Add 22 offline canary/benchmark examples, including stale baselines, dirty/wrong checkouts, CLI failures, cache regressions, and the synthetic benchmark generator/limits.

Validation

  • Full Windows/Ruby 4 suite: 582 examples, 0 failures, 8 platform-specific symlink skips; 98.9% line and 91.01% branch coverage (existing thresholds unchanged).
  • RuboCop: 258 files, no offenses. Documentation: 177 pages validated. Dependency-analysis reproducers pass; git diff --check clean.
  • All six pinned real-world runs pass locally: Rack 49 files / 133 or 140 internal edges; Rake 44 / 57 or 78; RSpec Core 74 / 18 or 157. Golden fingerprints match without --record. All three dedicated Linux canary jobs and the full normal CI matrix have now passed: https://github.com/LukasNiessen/ArchUnitRuby/actions/runs/39752135085

Scope and review

  • Independent sibling of Add fail-closed dependency vulnerability scanning #61; both build on Harden deep graph analysis, source boundaries, and release readiness #60, without requiring one another. Retarget after the underlying review stack merges. Nothing is merged or published here.
  • Production lib-only scope is consistent across hosts. Upstream security test fixtures are intentionally excluded; do not disable host antivirus to read blocked files. These are static regression checks, not a claim of upstream architectural defects or complete Ruby modeling.
  • --record emits proposed observations with baseline_verified=false and never rewrites the manifest. Baseline updates require review.

Combined-branch check

Locally combined this branch with #61 on an unpushed integration branch: release metadata passed, 593 examples with 0 failures (8 Windows symlink skips), 98.9% line / 90.81% branch coverage, and 260 RuboCop files clean. Both PRs remain separate and unmerged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant