PR 6.2: notebooks 03 (leakage + time windows) + 04 (lift / calibration / value / cohort)

[shaypal5]

Summary

• Notebook 03 — leakage and time windows: turns the documented total_touches_all trap into a teaching moment. Reads the trap label off the feature dictionary; proves the trap by construction via a same-table comparison of total_touches_all (full-horizon) vs touch_count (snapshot-safe) — mean post-snapshot delta 3.2 touches/lead, 82 % of leads positive. Standalone-AUC probe (~0.53, looks innocuous) ⇒ full-panel ± trap ablation showing HistGBM extracts +0.032 AUC where LR only squeezes +0.009. Pedagogical headline: standalone AUC probes undersell tree-friendly leakage.
• Notebook 04 — lift / calibration / value-aware ranking / cohort / bootstrap: four ranking lenses AUC alone misses. Reliability diagram, lift + cumulative gains, expected_acv × P(convert) value-aware ranking (top-50 ACV-capture jumps 0.16 → 0.40 vs P-only), threshold selection for fixed top-K capacity, cohort-shift evaluation that reproduces the validation report's cohort_shift.intermediate block exactly (0.8754 / 0.8908 / −0.0155), and a 200-iter bootstrap of the test set as the within-bundle confidence band — explicitly framed as the proxy for true cross-seed sweep, since public-bundle consumers can't rebuild bundles without leadforge installed.
• Audit-sync wiring + builder pattern: new cohort_shift.intermediate block in release/notebooks/_release_targets.json, new test_cohort_shift_targets_match_validation_report extension to the existing audit-sync test, both new builders inherit PR 6.1's deterministic-cell-ID + --out byte-stability pattern, both new notebooks added to the byte-stability and end-to-end execution test parametrisations.

Design decisions worth flagging

Notebook 03 framing pivot. The brief proposed an "engagement-only / firmographic-only feature panel ± trap" pedagogy on the assumption that the trap would dominate a thin set. Empirical AUCs say firmographic-only is at chance even with the trap (0.49–0.53), so the pedagogy doesn't hold. The reframed lesson — GBM extracts more from the trap than its standalone AUC predicts; LR doesn't — is honest, comes from the data, and produces a meaningful sign-aware tolerance gate (gbm_lift > 0.015).

Notebook 04 trap posture. Headline LR/GBM panel drops the trap (matches notebook 02; gives honest production numbers). Cohort-shift section keeps the trap to reproduce the report's published cohort-shift numbers exactly. Both postures are explained inline in their respective sections.

Bootstrap, not seed sweep. Resamples test-set indices 200× and recomputes already-scored AUC/AP — sub-second runtime, captures sampling noise on a single generated world but not generation-process noise across seeds. PR 6.1 forward-pointed to a "seed-sweep harness" but a true cross-seed sweep would require regenerating bundles, which Kaggle/HF consumers can't do; bootstrap is the honest within-bundle substitute and the notebook says so.

Cohort-shift reproduction. First implementation used COHORT_TRAIN_FRAC=0.7 and random_state=42; got cohort_split_auc=0.8854 (close but not equal to the report's 0.8908). Self-review caught the validator uses COHORT_TRAIN_FRAC=0.85 and model_random_state=0; switching matched the report to four decimal places.

Acceptance gates (from docs/release/v1_acceptance_gates.md)

• G13.1 — both notebooks execute top-to-bottom from a clean kernel in <10 s each (well under the 3-min budget).
• G13.2 — per-metric tolerances (AUC/Brier ±0.02, AP / top-decile ±0.05); cohort-shift values audit-synced against the validation report; bootstrap medians pinned inline against seed-42 single-run values.
• G13.3 — both notebooks assert manifest["exposure_mode"] == "student_public" and load only from release/intermediate/.

Test plan

• ruff check . → all checks passed
• ruff format --check . → 191 files already formatted
• mypy leadforge/ → no issues found in 82 source files
• pytest -x -q → 1260 passed, 5 publish-extra-gated skips, 1m58s
• pytest tests/scripts/test_release_notebook_builders.py tests/release/notebooks/ → 28 passed (byte-stability + nbclient execution + targets-match-report audit-sync)
• python scripts/probe_relational_leakage.py release/{intro,intermediate,advanced} --max-accuracy 0.65 → 0/3 paths flag rate on every tier
• python scripts/verify_hash_determinism.py → PASS 67/67
• python scripts/validate_release_candidate.py --no-rebuild → exits 0; rewrote validation_report.{json,md} timestamps which were reverted before commit
• BUNDLE_SCHEMA_VERSION unchanged at 5

Out of scope (PR 6.3)

• docs/release/break_me_guide.md — forward-pointed to as plain backtick-wrapped text in both notebooks (no dead Markdown link).
• .github/ISSUE_TEMPLATE/{dataset_breakage_report,realism_feedback}.yml
• docs/release/v2_decision_log.md
• release/README.md link update

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds the next two release notebooks (03 and 04) to the v1 educational notebook sequence, along with builder scripts and CI gates to keep the notebooks deterministic, executable, and audit-synced to the validation report.

Changes:

• Add Notebook 03 (leakage + time windows) and Notebook 04 (lift/calibration/value ranking/cohort shift/bootstrap), plus their deterministic builders.
• Extend notebook CI coverage: execute the new notebooks end-to-end and include their builders in byte-stability checks.
• Add a cohort_shift.intermediate targets block and a dedicated audit-sync test to ensure it matches validation_report.json.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/scripts/test_release_notebook_builders.py	Adds notebook 03/04 builders to the byte-stability + “matches committed” gate.
tests/release/notebooks/test_release_targets_match_report.py	Skips cohort_shift in the tier-median loop and adds a dedicated cohort-shift audit-sync test.
tests/release/notebooks/test_execute_notebooks.py	Adds notebook 03/04 to the nbclient execution matrix.
scripts/build_release_notebook_03.py	New deterministic builder that generates notebook 03 content.
scripts/build_release_notebook_04.py	New deterministic builder that generates notebook 04 content and pins metrics/tolerances.
release/notebooks/03_leakage_and_time_windows.ipynb	Generated notebook 03 committed artifact.
release/notebooks/04_lift_calibration_value_ranking.ipynb	Generated notebook 04 committed artifact.
release/notebooks/_release_targets.json	Adds cohort_shift.intermediate pinned values for notebook 04’s cohort-shift reproduction gate.
.agent-plan.md	Marks Phase 6.2 notebook work as complete and documents the delivered content.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

pr-agent-context report:

This run includes unresolved review comments on PR #74 in repository https://github.com/leadforge-dev/leadforge

For each unresolved review comment, recommend one of: resolve as irrelevant, accept and implement
the recommended solution, open a separate issue and resolve as out-of-scope for this PR, accept and
implement a different solution, or resolve as already treated by the code.

After I reply with my decision per item, implement the accepted actions, resolve the corresponding
PR comments, and push all of these changes in a single commit.

# Copilot Comments

## COPILOT-1
Location: scripts/build_release_notebook_04.py
URL: https://github.com/leadforge-dev/leadforge/pull/74#discussion_r3200276155
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    Grammar: “reflect a honest production model” should be “reflect an honest production model”.

## COPILOT-2
Location: scripts/build_release_notebook_04.py
URL: https://github.com/leadforge-dev/leadforge/pull/74#discussion_r3200276229
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    The cohort-shift description says “train on the first 70% … score the last 30%”, but immediately below (and in the code) the split is 85%/15% (COHORT_TRAIN_FRAC=0.85). Please update the prose so it matches the implemented/validated split to avoid confusing readers and to keep the “mirrors the validator” claim precise.

## COPILOT-3
Location: scripts/build_release_notebook_04.py:433
URL: https://github.com/leadforge-dev/leadforge/pull/74#discussion_r3200276269
Root author: copilot-pull-request-reviewer

Comment:
    The comment says the K-th highest probability threshold “admits exactly K leads (ties resolved by score order)”, but using `mask = lr_probs >= threshold` can admit more than K when there are ties at the threshold, and there’s no explicit tie-break. Either adjust the comment/prose, or implement a true top-K selection (e.g., via argsort indices) if you want exactly CAPACITY leads.

## COPILOT-4
Location: scripts/build_release_notebook_04.py:664
URL: https://github.com/leadforge-dev/leadforge/pull/74#discussion_r3200276306
Root author: copilot-pull-request-reviewer

Comment:
    Bootstrap loop: the code comment says “Degenerate resample — re-roll”, but the implementation writes NaNs and continues. Since the arrays are later used for quantiles/histograms, it’d be more robust to actually re-draw indices until non-degenerate (or explicitly filter finite values everywhere and handle the all-NaN case).

## COPILOT-5
Location: scripts/build_release_notebook_04.py
URL: https://github.com/leadforge-dev/leadforge/pull/74#discussion_r3200276331
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    Summary numbers appear inconsistent with the pinned/printed metrics above (e.g., max bin error is pinned at 0.1344 in NB04_TARGETS and lift@10% is ~2.75, but the summary says ≈0.19 and ~2.6×). Please update the summary text so it reflects the actual computed values for this notebook run.

## COPILOT-6
Location: release/notebooks/_release_targets.json
URL: https://github.com/leadforge-dev/leadforge/pull/74#discussion_r3200276360
Status: outdated
Root author: copilot-pull-request-reviewer

Comment:
    The top-level `_doc` says this file contains only “Cross-seed-median metric values”, but the new `cohort_shift` block is explicitly single-seed (seed 42). Consider updating the top-level docstring to reflect that the file now contains a mix of cross-seed medians (per-tier metrics) and single-seed cohort-shift metrics, so downstream readers don’t assume everything here is a median.

Run metadata:

Tool ref: v4
Tool version: 4.0.21
Trigger: commit pushed
Workflow run: 25491133072 attempt 1
Comment timestamp: 2026-05-07T10:46:00.887507+00:00
PR head commit: 397feb544b3314f235a58cd5bb73bed480ba652f