Add pinned real-world static canaries and benchmark regression tests

.github/workflows/canary.yml

@@ -0,0 +1,76 @@
+name: Real-world static canary
+
+on:
+  workflow_dispatch:
+  pull_request:
+    paths:
+      - 'lib/**'
+      - 'Gemfile'
+      - 'Gemfile.lock'
+      - 'benchmark/extraction.rb'
+      - 'benchmark/real_world_canary.rb'
+      - 'benchmark/canary/**'
+      - 'spec/benchmark/**'
+      - '.github/workflows/canary.yml'
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  canary:
+    name: Static lib analysis — ${{ matrix.name }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - name: rack
+            repository: rack/rack
+            revision: e1f22fdbe99afd2126b6fbf05bb12399359574b7
+          - name: rake
+            repository: ruby/rake
+            revision: 15df62f11e7f3980bb404bf1471e326cb7ad40ec
+          - name: rspec-core
+            repository: rspec/rspec-core
+            revision: aec5f49485d97908183dbe790a7fefb8baaa8091
+    defaults:
+      run:
+        working-directory: archunit
+    steps:
+      - name: Check out ArchUnitRuby
+        uses: actions/checkout@v7
+        with:
+          path: archunit
+          persist-credentials: false
+      - name: Check out pinned corpus as data only
+        uses: actions/checkout@v7
+        with:
+          repository: ${{ matrix.repository }}
+          ref: ${{ matrix.revision }}
+          path: corpus
+          fetch-depth: 1
+          persist-credentials: false
+      - name: Set up this project's Ruby and dependencies only
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: '4.0'
+          bundler-cache: true
+          working-directory: archunit
+      - name: Validate baseline, cache reuse, and passing/failing rules
+        run: >-
+          bundle exec ruby benchmark/real_world_canary.rb
+          --project ${{ matrix.name }} --root ../corpus
+          --output pkg/canary-${{ matrix.name }}.json
+      - name: Retain performance and correctness evidence
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: canary-${{ matrix.name }}-${{ github.run_id }}
+          path: archunit/pkg/canary-${{ matrix.name }}.json
+          if-no-files-found: warn
+          retention-days: 14

README.md

@@ -588,6 +588,11 @@ bundle exec ruby benchmark/extraction.rb
 See the [benchmark guide](https://github.com/LukasNiessen/ArchUnitRuby/blob/main/benchmark/README.md)
 for corpus controls, JSON output, and CI limits.
 
+The separate [real-world canary](https://github.com/LukasNiessen/ArchUnitRuby/blob/main/benchmark/canary/README.md)
+checks pinned Rack, Rake, and RSpec production-library sources in both dependency modes, including
+passing and deliberately failing rules, graph baselines, and cold/warm performance. It never runs
+those projects or installs their dependencies, and is not scheduled automatically yet.
+
 The implementation conventions and intended dependency directions live in [`AGENTS.md`](AGENTS.md).
 
 ## 📅 Plans and Current Limitations

benchmark/README.md

@@ -1,5 +1,8 @@
 # Extraction performance benchmark
 
+For pinned production-library compatibility checks, public passing/failing rule witnesses, and
+real-world timing artifacts, see the [static canary guide](canary/README.md).
+
 The benchmark generates a deterministic multi-gem Ruby repository, then measures one cold graph
 build and one same-process cached lookup. Target application code and generated gemspecs are never
 executed.

benchmark/canary/README.md

@@ -0,0 +1,70 @@
+# Real-world static canary
+
+The canary statically analyzes the **production `lib` directory only** of three pinned Ruby
+repositories. It does not install their dependencies, evaluate their Gemfiles or gemspecs, execute
+their source, or run their tests. It is not an upstream architecture-quality judgment.
+
+| Corpus | Pinned revision | Ruby files | Internal edges: imports / expanded |
+| --- | --- | ---: | ---: |
+| [Rack](https://github.com/rack/rack) 3.2.6 | `e1f22fdbe99afd2126b6fbf05bb12399359574b7` | 49 | 133 / 140 |
+| [Rake](https://github.com/ruby/rake) snapshot | `15df62f11e7f3980bb404bf1471e326cb7ad40ec` | 44 | 57 / 78 |
+| [RSpec Core](https://github.com/rspec/rspec-core) snapshot | `aec5f49485d97908183dbe790a7fefb8baaa8091` | 74 | 18 / 157 |
+
+These are reviewed baselines recorded on 2026-10-22, not moving branch names. `projects.json`
+records the full commits, a known import witness, and per-mode file/edge counts and graph hashes.
+The workflow matrix must match that manifest; an offline contract test enforces it.
+
+## What it proves
+
+For both `imports` and `imports_and_constants`, the canary:
+
+- Confirms the exact checkout revision and a clean `lib` source scope.
+- Measures cold extraction, warm cache reuse, stage timings, and available heap/RSS counters.
+- Compares file counts, internal/external edge counts, and a sorted internal graph fingerprint.
+- Runs a passing public rule, then deliberately forbids one known real import and requires exactly
+  one `FileDependencyViolation` and an assertion failure containing the rule rationale.
+
+For example, the Rack witness is `rack.rb -> rack/builder.rb`, relative to its `lib` root. Forbidding
+this existing edge is an intentionally failing **test policy**, not evidence that Rack's design is wrong.
+Unknown dynamic imports, unresolved constants, and ambiguous names remain visible in profile counters;
+the canary does not claim complete modeling of Ruby or framework autoloading.
+
+## Run it
+
+After cloning a corpus and checking out the exact manifest commit:
+
+~~~bash
+bundle exec ruby benchmark/real_world_canary.rb \
+  --project rack --root /path/to/rack \
+  --output pkg/canary-rack.json
+~~~
+
+Corpus dependencies are never needed. The script uses ArchUnitRuby's own locked parser dependencies.
+Do not run `bundle install` or build tasks inside the corpus. Dirty or untracked files under `lib`
+are rejected; unrelated upstream test fixtures are outside the analysis scope. Some security test
+fixtures can trigger host antivirus products. Do not disable those protections or attempt alternate
+reads of blocked files: keep the declared source scope consistent across platforms.
+
+The separate **Real-world static canary** workflow runs on relevant pull requests and supports manual
+dispatch once merged. It is intentionally not scheduled yet. It uses read-only repository access,
+three isolated ten-minute jobs, and 14-day JSON artifacts, including error evidence on failures.
+Keep it advisory rather than adding it to required branch checks until its cost and stability are reviewed.
+Failures remain red; they are not hidden with `continue-on-error`.
+
+There are broad 120-second cold / 5-second warm ceilings per mode, plus the job timeout. Those are
+runaway guards, not performance promises. Local cold times for this lib-only corpus were roughly
+0.1–0.4 seconds per mode on Ruby 4.0.6 / Windows x64; compare repeated like-for-like samples before
+claiming a speedup. These small libraries are not substitutes for a full Rails monorepo benchmark.
+
+## Updating a baseline
+
+1. Choose and review a new full upstream commit. Update the workflow matrix and manifest together.
+2. Inspect the known witness in that source. Do not replace it with an arbitrary edge simply to pass.
+3. Run the script with `--record` to produce **proposed observations**. This option never rewrites the
+   manifest and marks its report `baseline_verified: false`.
+4. Review file/edge/hash differences and unresolved/ambiguous counters. If the extractor changed,
+   determine whether differences are intended or a regression before accepting new expectations.
+5. Edit the reviewed `modes` values in `projects.json`, rerun without `--record`, and verify Linux CI.
+
+Regular CI never records or automatically accepts a new baseline. The ordinary unit suite uses
+small synthetic projects and mocked git inspection; it does not clone these repositories or need a network.

benchmark/canary/projects.json

@@ -0,0 +1,61 @@
+{
+  "projects": {
+    "rack": {
+      "repository": "rack/rack",
+      "revision": "e1f22fdbe99afd2126b6fbf05bb12399359574b7",
+      "witness": { "source": "rack.rb", "target": "rack/builder.rb" },
+      "modes": {
+        "imports": {
+          "file_count": 49,
+          "internal_dependencies": 133,
+          "external_dependencies": 28,
+          "internal_graph_sha256": "8749f31622cfb15062f77de0f32eb870b5eb6f782bdc3365604b62236712510c"
+        },
+        "imports_and_constants": {
+          "file_count": 49,
+          "internal_dependencies": 140,
+          "external_dependencies": 28,
+          "internal_graph_sha256": "a3897a8c67acee0a1d372c5bb97dcfd7a862048e5759d8f3fb1507f3121498d6"
+        }
+      }
+    },
+    "rake": {
+      "repository": "ruby/rake",
+      "revision": "15df62f11e7f3980bb404bf1471e326cb7ad40ec",
+      "witness": { "source": "rake.rb", "target": "rake/application.rb" },
+      "modes": {
+        "imports": {
+          "file_count": 44,
+          "internal_dependencies": 57,
+          "external_dependencies": 12,
+          "internal_graph_sha256": "1b188eb049249b6f12cbb8d3307c6c3d4ad786e29a19d4a4f1b31f31fe3dae04"
+        },
+        "imports_and_constants": {
+          "file_count": 44,
+          "internal_dependencies": 78,
+          "external_dependencies": 12,
+          "internal_graph_sha256": "a2bcb171c24678a6a1277fe4b3b4b32afef5c23ba3f8b6ca263db7b7ee41e466"
+        }
+      }
+    },
+    "rspec-core": {
+      "repository": "rspec/rspec-core",
+      "revision": "aec5f49485d97908183dbe790a7fefb8baaa8091",
+      "witness": { "source": "rspec/autorun.rb", "target": "rspec/core.rb" },
+      "modes": {
+        "imports": {
+          "file_count": 74,
+          "internal_dependencies": 18,
+          "external_dependencies": 34,
+          "internal_graph_sha256": "81e0621a26487d438391595a7407d651b15025247caa37c20f1bb7fab77f8415"
+        },
+        "imports_and_constants": {
+          "file_count": 74,
+          "internal_dependencies": 157,
+          "external_dependencies": 34,
+          "internal_graph_sha256": "34f388962ada484dc84d055dd261cfadc16f17d5538dec2ea45de821f1ee0a4c"
+        }
+      }
+    }
+  }
+}