Skip to content

[C2] Receipt-gated parity/latency tests always skip in CI (green proves no parity) #292

Description

@DsThakurRawat

Labels: ci, parity, testing, priority:high
File: tests/test_decomposed_per_step_parity.py, _overhead.py, _e2e_latency.py, test_pro_consent.py — each pytest.skip("No receipt …"); receipts produced by scripts/modal_per_step_*.py, never run in pytest.yml

Description. The tests that assert cos≈1.0 parity, IOBinding overhead, and e2e latency budgets pytest.skip when their Modal receipt file is absent — which is always on a GitHub runner.

Why it matters. The gates that would catch an action-parity or latency regression are structurally guaranteed to skip in CI. A regression that breaks the product's core parity claim merges fully green.

Tasks.

  • Add a @pytest.mark.receipt marker so their skipped status is visible (not blended into normal skips).
  • Run the Modal jobs in a scheduled/pre-release workflow that uploads receipts as artifacts consumed by the pytest step.
  • Make release branches xfail/fail-loud when receipts are absent.

Dedup. #52 is STRICT_GATE for the finetune drop-gate; PR #231 makes that gate fail-closeable. Neither touches these receipt-gated parity tests. Distinct.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions