Plan v4 lead scoring dataset + leadforge engine roadmap

[shaypal5]

Summary

Comprehensive planning PR for the v4 lead scoring dataset — a pedagogically improved single-CSV dataset for an intro ML course. This PR contains only documentation and planning artifacts (no code changes, all 590 tests pass).

Why v4?

v1–v3 each fixed critical issues but revealed new ones:

• v1: label leakage via funnel_stage containing closed_won/closed_lost
• v2: 90-day snapshot = post-mortem classification (not prediction); deterministic binary proxies (reached_sql=0 → 0% conversion)
• v3: day-21 snapshot + clean proxies, but AUC only 0.62 due to flat category effects at intro difficulty

v4 addresses all of these and adds:

• Value-aware ranking (expected_acv feature)
• Temporal momentum (touches_week_1)
• Structured missingness (conditional on lead source, not just MCAR)
• Deliberate leakage trap (total_touches_all — full 90-day window, for classroom discussion)
• Stronger category signal via category_effect_scale in difficulty profiles

Milestones

Milestone	Scope	Status
v4-M0	Requirements, contract, agent instructions	✅ This PR
v4-M1	Engine: category signal tuning + windowed snapshots	⬜ Next
v4-M2	Build pipeline: snapshot builder + validation	⬜
v4-M3	Documentation + dataset release	⬜

Existing roadmap items — triage

Item	Decision	Rationale
M12: CLI --json/--strict flags	Deferred	No consumer needs it; low priority vs dataset
M14: Sample bundle generation	Absorbed into v4-M3	v4 dataset IS the sample
M14: Lead-scoring baseline notebook	Deferred	v4 validation script covers this
M14: Notebooks 3–4 (public/instructor, recipe customization)	Discarded	No current audience
M15: Docs polish + v1.0 RC	Deferred	Do after v4 ships

v4 dataset acceptance criteria

• 1,000 rows × 18 columns (17 features + 1 target)
• 30% conversion rate
• No deterministic groups (n≥50 at 0% or 100%)
• Baseline LR AUC 0.65–0.90 (without leakage trap)
• Leakage trap boosts AUC by ≥0.03
• Structured missingness (source-conditional, not just MCAR)
• Reproducible with seed 42

Files in this PR

• docs/v4/lead_scoring_v4_requirements.md — 7 requirements with rationale
• docs/v4/dataset_contract.md — schema contract, temporal gates, missingness patterns
• docs/v4/engine_changes_spec.md — what changes in the engine and where
• docs/v4/validation_spec.md — 8 mandatory + 3 warning checks
• docs/v4/implementation_plan.md — milestone breakdown with acceptance criteria
• Updated CLAUDE.md — repo map, generation workflow, student_public invariants
• Updated AGENTS.md — v4 implementation guide, coding conventions, testing
• Updated .agent-plan.md — v4 as next work, M12–M15 triage

What this PR does NOT do

• No code changes to the engine or any Python module
• No implementation of v4-M1 through v4-M3 (deferred to follow-up PRs)
• No changes to the simulation loop
• No new dependencies

🤖 Generated with Claude Code

[github-actions]

pr-agent-context report:

No unresolved review comments, failing checks, or actionable patch coverage gaps were found on PR
#19. Treat this PR as all clear unless new signals appear.

Run metadata:

Tool ref: v4
Tool version: 4.0.20
Trigger: pull request opened
Workflow run: 25109426149 attempt 1
Comment timestamp: 2026-04-29T12:41:56.785853+00:00
PR head commit: 1f45df09976b1cf10e5cca6a0a366a3270ff4364

[Copilot]

Pull request overview

Planning and documentation for the v4 lead scoring “intro” dataset, including requirements, dataset contract, validation spec, and an engine/build roadmap to address leakage, temporal framing, and pedagogical signal strength.

Changes:

• Added v4 planning docs: requirements, dataset contract, engine changes spec, validation spec, and milestone implementation plan.
• Updated contributor-facing guidance (CLAUDE.md, AGENTS.md) with repo map, workflow/commands, and v4 implementation checklist.
• Updated .agent-plan.md to prioritize v4 and triage older roadmap items.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 18 comments.

Show a summary per file

File	Description
docs/v4/validation_spec.md	Defines mandatory + warning validation checks for the v4 single-CSV dataset.
docs/v4/lead_scoring_v4_requirements.md	Captures v4 goals, rationale from v1–v3, and the proposed v4 column set + acceptance criteria.
docs/v4/implementation_plan.md	Milestone breakdown (M0–M3) and deliverables/acceptance criteria for v4 delivery.
docs/v4/engine_changes_spec.md	Specifies intended engine changes (difficulty/category signal tuning, snapshot windowing) and build-script responsibilities.
docs/v4/dataset_contract.md	Defines snapshot/target temporal gates, missingness expectations, subsampling, and reproducibility contract.
CLAUDE.md	Adds repo map + generation/validation workflow documentation and links to v4 docs.
AGENTS.md	Adds v4 implementation guidance, conventions, and a validation checklist.
.agent-plan.md	Reorients “Next Up” to v4 milestones and defers/triages older roadmap items.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

R7 specifies “Each column with nulls: 1–15% missing”, but validation_spec.md only enforces “no column > 20% missing” and dataset_contract.md includes structural missingness for days_since_last_touch (which may exceed 15% depending on touch sparsity). Consider relaxing/aligning this requirement (e.g., per-column max only, or explicit bounds per affected column).

Suggested change

	| Missingness per column | Each column with nulls: 1–15% missing |
	| Missingness per column | For nullable columns, no column may exceed 20% missing unless explicitly exempted below |
	| Missingness exceptions | `days_since_last_touch`: structural missingness allowed in [5%, 35%] because leads with no touches have natural NaN; `web_sessions`: overall missingness ≤20% |

[Copilot]

The “New features to compute” list includes days_since_first_touch, but docs/v4/lead_scoring_v4_requirements.md’s target column set does not include this feature (and fixes the dataset at 18 columns). Please reconcile whether days_since_first_touch is part of v4 (and update the schema/validation) or remove it here to keep the specs aligned.

Suggested change

	| `days_since_first_touch` | `snapshot_day - first_touch_day` (NaN if no touches) | Lead age signal |
	| `expected_acv` | Opportunity ACV if opp created by snapshot; else employee_band midpoint | Value feature |
	| `total_touches_all` | Count of ALL touches over full horizon (ignoring snapshot gate) | Leakage trap |
	### Files affected
	- `leadforge/render/snapshots.py` — add `snapshot_day` parameter, windowed filtering, new feature computations
	- `leadforge/schema/features.py` — add new `FeatureSpec` entries for the new columns
	| `expected_acv` | Opportunity ACV if opp created by snapshot; else employee_band midpoint | Value feature |
	| `total_touches_all` | Count of ALL touches over full horizon (ignoring snapshot gate) | Leakage trap |
	### Files affected
	- `leadforge/render/snapshots.py` — add `snapshot_day` parameter, windowed filtering, new feature computations
	- `leadforge/schema/features.py` — add new `FeatureSpec` entries for the added v4 columns

[Copilot]

“Any column containing _id suffix” is ambiguous (contains vs endswith). If the intent is to ban opaque identifiers like lead_id/account_id, consider clarifying to “any column name ending with _id” to avoid mis-implementing the check.

Suggested change

	- Any column containing `_id` suffix (opaque identifiers, not features)
	- Any column name ending with `_id` (opaque identifiers, not features)

[Copilot]

Structured missingness refers to total_touches == 0, but R5 requires dropping total_touches from v4. To avoid confusion, define the condition in terms of retained columns (e.g., inbound_touches + outbound_touches == 0) or explicitly note that total_touches is computed internally and not included in the final CSV.

Suggested change

	1. **Natural (structural):** `days_since_last_touch` is NaN when `total_touches == 0` (no touches recorded). Already exists but must be preserved.
	1. **Natural (structural):** `days_since_last_touch` is NaN when `inbound_touches + outbound_touches == 0` (no touches recorded). Already exists but must be preserved.

[Copilot]

The missingness contract uses total touches == 0 as the condition for structural NaNs in days_since_last_touch, but v4 drops total_touches from the final CSV. It would be clearer to express the condition using columns that remain in v4 (e.g., inbound_touches + outbound_touches == 0) or explicitly state that the condition is evaluated pre-drop during build.

Suggested change

	| `days_since_last_touch` | Structural | Natural | NaN when `total touches == 0` by snapshot |
	| `days_since_last_touch` | Structural | Natural | NaN when `inbound_touches + outbound_touches == 0` by snapshot |

[Copilot]

This plan says v4 is delivered across “4–6 PRs”, but later the timeline section states “The work is 4 PRs”. Please make the expected PR count consistent so the milestone planning is unambiguous.

Suggested change

	This plan implements the v4 lead scoring dataset in 4 milestones across 4–6 PRs. Each milestone produces testable artifacts and has explicit acceptance criteria.
	This plan implements the v4 lead scoring dataset in 4 milestones across 4 PRs. Each milestone produces testable artifacts and has explicit acceptance criteria.

[Copilot]

This section states the v4 build/validation scripts live in the leadforge repo (not datasets-private), but CLAUDE.md’s “Related repos” + workflow example place these scripts in leadforge-datasets-private. Please align the docs on a single canonical home (or document how scripts are shared) to avoid contributors building in the wrong repo.

Suggested change

	- `scripts/build_v4_snapshot.py` (new) — snapshot builder with missingness + leakage trap
	- `scripts/validate_v4_dataset.py` (new) — dataset-level validation
	- These live in the leadforge repo (not datasets-private)
	**v4-M3 (release):**
	- Work in `leadforge-datasets-private` repo
	- Work in `leadforge-datasets-private` repo
	- `scripts/build_v4_snapshot.py` (new) — snapshot builder with missingness + leakage trap
	- `scripts/validate_v4_dataset.py` (new) — dataset-level validation
	**v4-M3 (release):**
	- Continue in `leadforge-datasets-private` repo for dataset generation and release artifacts

[Copilot]

The implementation order diagram says “v4-M0 (planning PR — already done)”, but in this PR v4-M0 is the work being proposed/merged. Consider rephrasing to something time-invariant (e.g., “v4-M0: planning (this PR)”) to avoid the docs disagreeing with .agent-plan.md.

Suggested change

	v4-M0 (planning PR — already done)
	v4-M0: planning (this PR)

[Copilot]

“All negatives retained (up to 700); positives downsampled.” is a bit self-contradictory for a 1,000-row stratified sample (you can’t retain all negatives from a 5,000-lead bundle if there are more than 700). Consider rewording to specify the intended sampling policy (e.g., fixed 700 negatives sampled uniformly/stratified, and 300 positives downsampled as needed) and what happens if the source bundle’s base rate is below 30%.

Suggested change

	- Stratified subsampling to 1,000 rows at ~30% conversion rate.
	- All negatives retained (up to 700); positives downsampled.
	- Subsampling preserves within-class feature distributions.
	- Stratified subsampling produces 1,000 rows with a target mix of 700 negatives and 300 positives when the source bundle contains at least that many examples of each class.
	- Sampling policy: draw negatives uniformly at random without replacement up to 700 rows, and draw positives uniformly at random without replacement up to 300 rows.
	- If the source bundle has fewer than 300 positives, retain all available positives and fill the remaining rows with additional uniformly sampled negatives; if it has fewer than 700 negatives, retain all available negatives and fill the remaining rows with additional uniformly sampled positives.
	- Because sampling is uniform within each class, the subsample is intended to preserve within-class feature distributions up to normal sampling variation.

[Copilot]

R3 says missingness injection belongs in the snapshot builder (engine), but docs/v4/engine_changes_spec.md specifies missingness should be injected in the v4 build script (post-snapshot) to keep engine output clean. Please align R3 with the chosen design so implementation guidance is consistent.

Suggested change

	**Engine change needed:** Missingness injection in the snapshot builder, conditioned on feature values.
	**Implementation note:** Preserve natural structural missingness in the engine output, but inject the conditional missingness patterns in the v4 build script **after** snapshot generation, conditioned on feature values, to keep the snapshot builder output clean and reusable.