feat: wire difficulty profiles into simulation engine#52
Merged
Conversation
The simulation engine now modulates behavior based on difficulty profile parameters (signal_strength, noise_scale, missing_rate, outlier_rate, conversion_rate_range, committee_friction). Previously all three tiers produced identical ~70% conversion; now intro targets 30-45%, intermediate 18-28%, and advanced 8-15%. Key changes: - DifficultyParams dataclass carries numeric profile parameters - assign_mechanisms() uses per-motif calibration to compute target daily hazard rates from conversion_rate_range - signal_strength scales LatentScore weights - committee_friction modulates churn rate - build_snapshot() injects Gaussian noise, MCAR missingness, and outliers based on noise_scale/missing_rate/outlier_rate - check_difficulty_ordering() now validates actual conversion rates Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
shaypal5
added
type: feature
This comment has been minimized.
This comment has been minimized.
- Extract motif calibration to module-level constant with clear docs - Make _apply_difficulty_distortions() pure (copies input, no mutation) - Use LEAD_SNAPSHOT_FEATURES spec for column eligibility (not runtime dtype sniffing) — prevents accidental distortion of categoricals - Raise InvalidRecipeError on missing profile keys instead of silent defaults that mask typos - Non-uniform signal_strength scaling: primary weight × s, secondary weights × s^1.5 — reduces discriminability rather than just shifting the sigmoid - Use 5σ for outlier injection (vs 3σ) to be distinguishable from natural variation - Tests use tmp_path fixture instead of hardcoded /tmp paths - Add direct test of _apply_difficulty_distortions() verifying noise changes values and function is pure Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
pr-agent-context report: No unresolved review comments, failing checks, or actionable patch coverage gaps were found on PR #52 in repository https://github.com/leadforge-dev/leadforge. Treat this PR as all clear unless new signals appear.Run metadata: |
4 tasks
shaypal5
added a commit
that referenced
this pull request
May 3, 2026
…tion (#53) * fix: update release docs for difficulty modulation and regenerate bundles Remove stale "Known limitations" section claiming difficulty tiers share the same conversion rate. Replace with a "Difficulty tiers" section documenting actual conversion ranges. Regenerated all 4 release bundles with the difficulty-aware engine (PR #52): intro 41.5%, intermediate 20.1%, advanced 7.9%. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: consolidate difficulty info, update HF card, restore determinism task - Remove duplicate "Difficulty tiers" section from README; merge conversion rate (target + observed) into the existing Dataset summary table. Single source of truth, no duplication. - Remove hardcoded ranges from intro paragraph to reduce staleness risk. - Add conversion rate row to HF_DATASET_CARD.md summary table and update difficulty bullet — card was inconsistent with README. - Restore SHA-256 determinism verification as an unchecked Phase 5 task in .agent-plan.md (was silently dropped in previous commit). - Point to difficulty_profiles.yaml as source of truth for target ranges. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
check_difficulty_ordering()validator is no longer a no-op — it verifies actual ratesChanges
leadforge/core/models.pyDifficultyParamsfrozen dataclass + field onGenerationConfigleadforge/mechanisms/policies.pyconversion_rate_range;signal_strengthscales LatentScore weightsleadforge/simulation/engine.pydifficulty_params; churn rate modulated bycommittee_frictionleadforge/render/snapshots.py_apply_difficulty_distortions()injects noise/missingness/outliersleadforge/api/generator.pyDifficultyParamsfrom profile YAMLleadforge/api/bundle.pybuild_snapshot()leadforge/validation/difficulty.pytests/test_difficulty_modulation.pyCalibration results (20 seeds)
Test plan
python scripts/build_public_release.pyto verify three tiers produce visibly different datasets🤖 Generated with Claude Code