PR 4.1: channel-signal audit + release-grade dataset card

[shaypal5]

Summary

PR 4.1 — the only PR in Phase 4 of the v1 dataset release sequence. Documentation hardening on top of the leakage-safe bundles delivered in Phase 2 and the validation panel delivered in Phase 3. Adds one analysis script and four prose deliverables; no engine, render, or validation behaviour changes.

Roadmap: docs/release/v1_release_roadmap.md §"Phase 4 — PR 4.1".

What lands

1. scripts/audit_channel_signal.py (new) + 18 tests. For each tier and each of lead_source / first_touch_channel, computes per-channel conversion rate (train split) plus two univariate AUCs:

   • In-sample AUC: train rates → train labels (the historical 1-D Bayes interpretation; biased upward).
   • Out-of-sample AUC: train rates → held-out test labels — directly comparable to the source_only HistGBM baselines in release/validation/validation_report.json. The OOS numbers reproduce the validation-report cells for seed 42 exactly (intro 0.5014, intermediate 0.5139, advanced 0.5226).

   The audit does not assign a categorical "weak / moderate / strong" verdict and does not make a hard comparison between v1's 90-day closed-won label and the G2 / Gemini v2 industry MQL→SQL benchmark band — those measure different funnel transitions. The benchmark band is reproduced for context with an explicit caveat; the audit's Discussion section states facts and lets the reader compare.

2. docs/release/channel_signal_audit.md + sibling JSON (new) — auto-generated audit. v1's channel signal is empirically weak (per-channel rate spread ≤ 0.05, OOS univariate AUC ≈ 0.50–0.52 across all tiers), as expected: the simulator drives conversion through motif-family hazards keyed off latent traits, not channel-conditional probabilities. Channel-conditional encoding stays scoped post-v1.

3. docs/release/generation_method.md (new) — standalone DGP summary written for external readers. Reads alone, references docs/leadforge_architecture_spec.md. Covers the five generation layers, public-vs-instructor split, calibration / validation, and an explicit "what this is not" boundary. Satisfies G10.2.

4. docs/release/feature_dictionary.md (new) — narrative companion to the per-bundle feature_dictionary.csv. Groups every public-mode column by analytical role, documents difficulty modulation parameters, modelling defaults (now consistent with the flat-CSV shape), and the deliberate total_touches_all trap. Satisfies G10.3.

5. release/README.md (substantial rewrite, ~228 lines) — release-grade dataset card per Datasheets-for-Datasets / Data Cards Playbook (G10.1). Macro framing with citation to gemini_v2_summary.md, simulation simplifications (linked to generation_method.md for detail), calibration documentation linking to validation_report.md, public-vs-instructor redaction with concrete column lists citing BANNED_LEAD_COLUMNS / BANNED_OPP_COLUMNS / BANNED_TABLES / SNAPSHOT_FILTERED_TABLES from leakage_probes.py, intended/out-of-scope use, known limitations (G7.4.4 + weak channel + flat AUC + small cohort gap), composition section, adversarial-framing pointer, maintenance plan.

Failure modes the test plan guards against

• A future engine change diverges lead_source and first_touch_channel without updating the feature-dictionary claim that they're identical in v1.
  → test_lead_source_equals_first_touch_channel_in_v1 (parametrized over intro/intermediate/advanced) walks every train/valid/test split and asserts equality. Fails loudly; forces a doc update.

• A future bundle regeneration changes the channel mix but the audit artifacts in docs/release/channel_signal_audit.{md,json} are not re-run.
  → test_committed_audit_artifacts_match_fresh_regeneration re-runs the audit against the committed release/ bundles and asserts byte-equality with the committed audit MD/JSON. Fails CI on drift.

• A sklearn upgrade silently changes ROC-AUC behavior for a saturated 1-D categorical scorer.
  → test_audit_channel_in_sample_auc_pair_counting locks the closed-form value (17/18) on a synthetic fixture; test_audit_channel_oos_auc_matches_in_sample_when_test_is_train checks the OOS code path; test_audit_channel_oos_auc_handles_unseen_test_categories guards the train-base-rate fallback for unseen test categories.

• A future PR adds a new channel column but the audit doesn't render it (or worse, renders an identical-looking duplicate).
  → test_render_markdown_collapses_identical_columns and test_render_markdown_renders_distinct_columns_separately cover both branches of the channel-grouping logic.

• A BANNED_LEAD_COLUMNS rename silently invalidates the README's redaction-policy table.
  Not gated by a test in this PR — the table cites the constant by name, so a grep-once-after-rename habit catches it. Tracked as a follow-up: a docs-vs-code consistency check that imports the constant and asserts the README mentions it.

• The validator regenerates validation_report.{json,md} with a fresh wall-clock timestamp.
  → I deliberately reverted the regenerated report files in this PR to keep the diff focused; this PR does not modify the validator and the reverted timestamp is the only field that drifted.

Test plan (commands)

• python scripts/audit_channel_signal.py --release-dir release/ runs end-to-end, writes the audit MD + JSON
• pytest tests/scripts/test_audit_channel_signal.py -q — 18 tests pass
• Full suite: pytest -q — 1175 passed
• ruff check . — passed
• ruff format --check . — 168 files already formatted
• mypy leadforge/ scripts/audit_channel_signal.py — no issues
• python scripts/probe_relational_leakage.py release/{intro,intermediate,advanced} --max-accuracy 0.65 — all tiers exit 0
• python scripts/verify_hash_determinism.py — PASS 67/67 files identical
• python scripts/validate_release_candidate.py --no-rebuild — exits 0 (3 tiers × 5 seeds, leakage findings: 0)
• No TBD-* strings re-introduced in docs/release/v1_acceptance_gates.md
• BUNDLE_SCHEMA_VERSION unchanged at 5 (documentation-only PR)

Self-review history

This PR was rebased on top of itself after a brutal self-review pass:

• Audit methodology was rewritten to compute the OOS univariate AUC, drop the categorical verdict bands and prose (which made a category-error comparison between 90-day-conversion and MQL→SQL rates), use a tuple-of-pairs immutable dataclass field for the industry benchmarks, and group identical channel columns in the markdown output. See commit ccab336.
• Doc cleanup: trimmed release/README.md from 434 to 228 lines, dropped the developer-facing module-paths table from generation_method.md, fixed a factually wrong is_leakage_trap=True claim in feature_dictionary.md, and reworded the modelling-defaults checklist to be consistent with the flat-CSV shape. See commit 61964a9.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a deterministic “channel-signal” audit plus release-grade documentation for the v1 dataset release, without changing engine/render/validation behavior. The main addition is a new analysis script (and tests) that generates a Markdown/JSON audit report, alongside substantial dataset-card hardening and two new release-facing prose docs.

Changes:

• Add scripts/audit_channel_signal.py to compute per-channel conversion rates and a deterministic univariate AUC, emitting docs/release/channel_signal_audit.{md,json}.
• Add test coverage for the audit logic, rendering, CLI error paths, and (when release/ is present) byte-identical determinism across repeated runs.
• Rewrite release/README.md into a release-grade dataset card and add docs/release/{generation_method,feature_dictionary}.md.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
scripts/audit_channel_signal.py	New deterministic channel-signal audit script with JSON/Markdown rendering + CLI.
tests/scripts/test_audit_channel_signal.py	New unit + CLI tests, including an optional determinism guard against committed release/ bundles.
docs/release/channel_signal_audit.md	Generated audit report (human-readable).
docs/release/channel_signal_audit.json	Generated audit report (machine-readable).
release/README.md	Major rewrite into a release-grade dataset card with calibration, redaction policy, limitations, and usage guidance.
docs/release/generation_method.md	New external-reader-friendly generation method overview.
docs/release/feature_dictionary.md	New narrative feature dictionary grouped by analytical role + modelling guidance.
.agent-plan.md	Roadmap checklist updated to mark Phase 4 PR 4.1 items complete.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

[github-actions]

pr-agent-context report:

This run includes unresolved review comments on PR #69 in repository https://github.com/leadforge-dev/leadforge

For each unresolved review comment, recommend one of: resolve as irrelevant, accept and implement
the recommended solution, open a separate issue and resolve as out-of-scope for this PR, accept and
implement a different solution, or resolve as already treated by the code.

After I reply with my decision per item, implement the accepted actions, resolve the corresponding
PR comments, and push all of these changes in a single commit.

# Copilot Comments

## COPILOT-1
Location: scripts/audit_channel_signal.py
URL: https://github.com/leadforge-dev/leadforge/pull/69#discussion_r3195378952
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    `_label_to_int()` only special-cases NumPy/pandas `bool` dtype, but snapshots/task splits in this repo use pandas nullable `boolean` dtype (see `tests/render/test_render.py` asserting `.dtype.name == "boolean"`). Consider aligning with the existing pattern used elsewhere (e.g., `series.astype("boolean").fillna(False).astype(int)`) so the audit script reliably handles the canonical label dtype and any unexpected nulls without `to_numeric(..., errors="raise")` failures.

## COPILOT-2
Location: release/README.md
URL: https://github.com/leadforge-dev/leadforge/pull/69#discussion_r3195379011
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    The Quick start example merges a Series (`touch_counts`) using `on="lead_id"`, but `touch_counts` has `lead_id` in its index (not a column). As written this snippet is likely to error or produce an unintended join. Convert `touch_counts` to a DataFrame/reset its index, or merge with `right_index=True` so the example is runnable as-is.

## COPILOT-3
Location: docs/release/feature_dictionary.md:71
URL: https://github.com/leadforge-dev/leadforge/pull/69#discussion_r3195379043
Root author: copilot-pull-request-reviewer

Comment:
    The firmographics section says "Six columns" but the table lists five (`industry`, `region`, `employee_band`, `estimated_revenue_band`, `process_maturity_band`). If the sixth is `account_id` (currently documented under identity), it would help to either include it explicitly here or adjust the count so the narrative matches the feature list.

## COPILOT-4
Location: docs/release/feature_dictionary.md
URL: https://github.com/leadforge-dev/leadforge/pull/69#discussion_r3195379081
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    Typo: "A bandage of the latent `process_maturity` trait" reads like a misspelling (likely "banding"/"banded"), and could confuse readers about what the column represents.

## COPILOT-5
Location: docs/release/feature_dictionary.md
URL: https://github.com/leadforge-dev/leadforge/pull/69#discussion_r3195813969
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    `touches_week_1` is described as covering days 0–6, but the snapshot builder counts touches where `_day <= 7` (days 0–7 inclusive). Please align the documentation with the implemented window (or adjust the implementation if 0–6 is intended) to avoid confusing readers about what the feature represents.

## COPILOT-6
Location: scripts/audit_channel_signal.py
URL: https://github.com/leadforge-dev/leadforge/pull/69#discussion_r3195814053
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    For byte-identical reproducibility across operating systems/locales, consider writing the markdown/JSON outputs with an explicit encoding (e.g., UTF-8). `Path.write_text()` uses the platform default encoding, which can change the output bytes on Windows and break the determinism/integrity guarantees you're testing for.

## COPILOT-7
Location: scripts/audit_channel_signal.py:439
URL: https://github.com/leadforge-dev/leadforge/pull/69#discussion_r3195814096
Root author: copilot-pull-request-reviewer

Comment:
    `render_markdown()` hard-codes links to `docs/release/channel_signal_audit.json` and `release/validation/validation_report.json`. Since the CLI supports `--out-json` and `--release-dir`, these links can become incorrect when running the script on a non-canonical directory. Consider wording this as “see the JSON output alongside this file” or derive the paths from the actual CLI args/report metadata.

## COPILOT-8
Location: scripts/audit_channel_signal.py
URL: https://github.com/leadforge-dev/leadforge/pull/69#discussion_r3195814127
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    The Discussion section includes a seed-specific claim (“For seed 42 the OOS numbers below match…”), but the report content is generated for whatever bundle `--release-dir` points at. If someone runs the script on a different seed/bundle family, this statement can become false. Consider removing the seed-specific assertion or computing/printing the bundle seed(s) from the manifests and interpolating that into the text.

Run metadata:

Tool ref: v4
Tool version: 4.0.21
Trigger: commit pushed
Workflow run: 25440138616 attempt 1
Comment timestamp: 2026-05-06T14:03:35.105557+00:00
PR head commit: f84c8b6fff8c7b127e14a25119f41b9ad7f92a0b

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.