test(setup): experimental end-to-end setup-flow matrix (all ecosystems, workspaces, monorepo) (#98)

* test(setup): add experimental end-to-end setup-flow matrix

Adds a non-blocking, data-driven test matrix that verifies the intended
`socket-patch setup` flow end to end for every supported
ecosystem/package manager:

  0. prepare a project with a dependency + a committed patch set
  1. run `socket-patch setup` to configure install hooks
  2. run the native install command for the package manager
  3. check whether the patch was applied (marker on disk)

plus negative controls (no setup, empty/wrong-target/alt patch sets).

The suite is ASPIRATIONAL and intentionally non-blocking: `setup` only
configures npm-family hooks today, so non-npm `baseline_with_setup`
cases are expected `known_gap`s — a baseline of what `setup` must
eventually support. Results are classified against a recorded baseline;
the runner exits non-zero only on a regression.

Components:
- tests/setup_matrix/matrix.json — declarative cases (targets x scenarios),
  the single source of truth for both the runner and the Rust wrappers.
- tests/setup_matrix/run-case.sh — self-contained bash flow driver
  (scaffold -> setup -> install -> verify -> JSON); generates the
  npx/pnpm shims inline so the hook resolves to the local binary instead
  of fetching the published wrapper.
- scripts/setup-matrix.sh — orchestrator (build/run/list/query/results),
  classifies pass/known_gap/progress/regression, emits machine-readable
  JSON.
- crates/socket-patch-cli/tests/setup_matrix_<eco>.rs (+ shared module),
  gated by a new `setup-e2e` feature; assert the aspirational ideal.
- tests/docker/Dockerfile.{npm,pypi} extended additively (pnpm/yarn via
  corepack; uv/poetry/pdm/hatch) — existing docker_e2e tests unaffected.
- ci.yml: a `setup-matrix` job, `continue-on-error: true` (must stay out
  of required checks).

No CLI/source behavior changes. Verified in Docker on all 9 images
(socket-patch 3.3.0): 80 cases, 56 pass / 24 known_gap, 0 regression /
0 error; npm/yarn/pnpm/bun apply, everything else is a documented gap;
all negative controls pass (no leaks).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(setup): add nested-workspace + polyglot-monorepo layouts

Extends the setup-flow matrix with an `SM_LAYOUT` dimension modelling
real-world deployments beyond a single project:

- workspace (npm, pnpm, yarn, pip, uv): a root + several members,
  including a deeply-nested member and one with no dependency on the
  patched package. Exercises `setup`'s workspace handling — npm/yarn
  write the hook to every member, pnpm only to the root — and the
  cross-workspace apply on a single root install. npm/pnpm/yarn apply
  (the dependency hoists / lands in the pnpm store and is patched once);
  pip (nested requirements) and uv (uv workspace, one shared .venv) are
  Python gaps.
- monorepo: a polyglot repo with an npm workspace alongside
  python/rust/go/php/ruby/nuget/deno manifests. Confirms `setup` works in
  a mixed environment — it configures the npm hooks and does not choke on
  the foreign manifests; a root `npm install` then patches the npm slice.
  Runs in the npm image; the foreign manifests are present to test
  setup's robustness, not installed.

Wiring: `matrix.json` gains workspace_targets/scenarios and
monorepo_targets/scenarios; `run-case.sh` gains layout-aware scaffold /
install / multi-target verification; `scripts/setup-matrix.sh` threads a
`layout` column (+ `query --layout`); the Rust harness gains
`run_workspace_pm` / `run_monorepo`, with `*_workspace` tests on the
npm/pypi wrappers and a new `setup_matrix_monorepo.rs`.

Real-world finding (and fix in the harness): the install hook's `apply`
must run with the package manager's per-script cwd — root for the
project, the member dir for each member — so member postinstalls find no
manifest and no-op while the root applies. The driver therefore does NOT
pin SOCKET_CWD; pinning it to the root makes every member apply target
the root manifest and fail mid-install with "no packages found on disk",
breaking `npm install` in a workspace.

Verified in Docker (socket-patch 3.3.0): npm/pnpm/yarn workspace and the
monorepo apply (pass); pip/uv workspace are known_gap; single-project
cases unchanged. 92 cases total; 0 regressions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(setup): add patch-missing ablation controls across all layouts

Adds a `patch_missing` ablation (new `patchset: none` — no `.socket/`
fixture committed) to the single, workspace and monorepo scenario sets,
complementing the existing setup-not-run controls. Together they are the
controls that confirm `setup` is correct: each is identical to the
corresponding `*_with_setup` case except for the single removed factor
(the setup step, or the committed patch), and each must run UNPATCHED.

So every "it applies" case is now flanked by both ablations, e.g. for
single npm:
  baseline_with_setup  -> applied   (patch + setup)
  no_setup_control     -> unpatched (setup ablated)
  patch_missing        -> unpatched (patch ablated)

`run-case.sh` skips the fixture entirely for `patchset: none` (so the
hook's apply finds no manifest and no-ops — distinct from `empty`, where
the manifest exists but lists zero patches). No orchestrator/Rust changes
needed; the scenarios are data-driven and picked up automatically.

Matrix grows to 114 cases. Verified in Docker (3.3.0): all 22
patch_missing cases pass (run unpatched) — single 16/16, workspace 5/5,
monorepo 1/1; with_setup cases still apply; 0 regressions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: mikolalysenko

.github/workflows/ci.yml

@@ -577,3 +577,75 @@ jobs:
 
       - name: Run ${{ matrix.ecosystem }} Docker e2e test
         run: cargo test -p socket-patch-cli --features docker-e2e --test docker_e2e_${{ matrix.ecosystem }}
+
+  # ----------------------------------------------------------------------
+  # Experimental `setup`-flow matrix (NON-BLOCKING).
+  #
+  # For each ecosystem/package manager, drives the full intended flow —
+  # prepare deps + a committed patch set, run `socket-patch setup`, run
+  # the native install, check whether the patch was applied — plus the
+  # negative controls (no setup, empty/wrong/alt patch sets). See
+  # tests/setup_matrix/ and scripts/setup-matrix.sh.
+  #
+  # This is EXPERIMENTAL and intentionally not required to pass yet:
+  # `setup` only configures npm-family install hooks today, so most
+  # non-npm `baseline_with_setup` cases are EXPECTED to fail (a baseline
+  # of what `setup` must eventually support). `continue-on-error: true`
+  # means this job never blocks a PR — it must ALSO be left OUT of the
+  # repo's required status checks (configured in the branch-protection
+  # UI, not in this file). The orchestrator exits non-zero only on a
+  # *regression* vs the recorded baseline; the full per-case result set
+  # is uploaded as a JSON artifact for inspection.
+  # ----------------------------------------------------------------------
+  setup-matrix:
+    runs-on: ubuntu-latest
+    continue-on-error: true
+    permissions:
+      contents: read
+    strategy:
+      fail-fast: false
+      matrix:
+        ecosystem: [npm, pypi, cargo, gem, golang, maven, composer, nuget, deno]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - name: Set up Docker Buildx
+        # `driver: docker` — the per-ecosystem image's `FROM
+        # socket-patch-test-base:latest` only resolves when buildx talks
+        # directly to the host docker daemon (see e2e-docker above).
+        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
+        with:
+          driver: docker
+
+      - name: Install Rust
+        run: rustup show
+
+      - name: Build base image
+        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
+        with:
+          context: .
+          file: tests/docker/Dockerfile.base
+          tags: socket-patch-test-base:latest
+          load: true
+
+      - name: Build ${{ matrix.ecosystem }} image
+        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
+        with:
+          context: .
+          file: tests/docker/Dockerfile.${{ matrix.ecosystem }}
+          tags: socket-patch-test-${{ matrix.ecosystem }}:latest
+          load: true
+
+      - name: Run ${{ matrix.ecosystem }} setup-matrix
+        run: scripts/setup-matrix.sh run --ecosystem ${{ matrix.ecosystem }} --out "report-${{ matrix.ecosystem }}.json"
+
+      - name: Upload ${{ matrix.ecosystem }} setup-matrix report
+        if: always()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: setup-matrix-${{ matrix.ecosystem }}
+          path: report-${{ matrix.ecosystem }}.json
+          if-no-files-found: warn

crates/socket-patch-cli/Cargo.toml

@@ -39,6 +39,16 @@ deno = ["socket-patch-core/deno"]
 # `tests/docker_e2e_*.rs`. Tests in this suite require either a running
 # Docker daemon OR `SOCKET_PATCH_TEST_HOST=1` (host-toolchain mode).
 docker-e2e = []
+# Enables the experimental `setup` end-to-end test matrix under
+# `tests/setup_matrix_*.rs`, which drives the `socket-patch setup` →
+# native-install → patch-applied flow across every ecosystem/package
+# manager via `tests/setup_matrix/run-case.sh`. Same runtime requirement
+# as docker-e2e (Docker daemon OR `SOCKET_PATCH_TEST_HOST=1`). These
+# tests are ASPIRATIONAL: they assert the ideal (install applies the
+# patch) and are EXPECTED to fail for ecosystems whose install hooks
+# `setup` does not yet configure. Kept off `--all-features`-required CI;
+# the dedicated `setup-matrix` CI job runs them non-blocking.
+setup-e2e = []
 
 [dev-dependencies]
 sha2 = { workspace = true }

crates/socket-patch-cli/tests/setup_matrix_cargo.rs

@@ -0,0 +1,14 @@
+//! setup-matrix: cargo ecosystem. `setup` is a no-op for Rust projects
+//! (no package.json) and cargo has no post-install hook, so the
+//! with-setup cases are an EXPECTED BASELINE GAP.
+//!
+//! Run: `cargo test -p socket-patch-cli --features setup-e2e --test setup_matrix_cargo`
+#![cfg(feature = "setup-e2e")]
+
+#[path = "setup_matrix_common/mod.rs"]
+mod smc;
+
+#[test]
+fn cargo() {
+    smc::run_pm("cargo", "cargo");
+}