Improve investing decision quality: constraints, EV neutrality, eval metrics#10
Merged
Merged
Conversation
…metrics Three fixes to the finance planner and backtest harness: - Honor per-goal risk constraints in sizing. drawdown_limit and risk_tolerance constraints were validated and warned about but never read; the risk budget came only from pack/policy config. A drawdown limit now caps the concentration and per-trade risk budgets, and a conservative/aggressive tolerance scales the Kelly fraction. The move rationale names what bound. - Stop rewarding volatility in the EV score. With a flat 0.5 win probability and winFrac = ratio*lossFrac, expected value grew monotonically with volatility, so the EV component favoured the most volatile ticker on every initial plan. The EV score is now neutral (0.5) until a signal tilts the odds, and a signal hint only informs the ticker it names. - Add decision-quality metrics to the backtest report: a Brier score of top-move confidence against realized forward returns, per-decision forward-return attribution, and a noise-robustness rate (share of non-kill events that did not trigger a replan). Forward returns are computed after the replay, so no future data reaches a decision.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Quick-wins pass on decision quality for the investing pack, found while evaluating the finance planner's scoring path.
What changed
Per-goal risk constraints now bind sizing (
internal/finance/constraints.go, new)The investing pack validates
drawdown_limit/risk_toleranceconstraints and even warns "sizing will fall back to conservative defaults" when they're missing — but they were never read; the risk budget came solely from pack/policy config.EffectiveRiskBudgetnow tightens the budget per goal:MaxPositionPct(a total loss of one position can't exceed the stated limit) andMaxPortfolioRiskPct(five consecutive stop-outs stay within it),EV no longer rewards volatility
With the default flat
winProb=0.5andwinFrac = RewardRiskRatio * lossFrac, expected value reduced to0.5*lossFrac*(ratio-1)— monotonically increasing in volatility, so every initial plan tilted toward the most volatile ticker. The EV component is now neutral (NeutralEVScore = 0.5) until a signal carries an actual probability hint. Relatedly, a signal hint now only informs the ticker it names (a valuation hint on ACME no longer lifted GLOBEX too); unnamed signals (e.g. macro) still apply to all candidates.Backtest harness measures decision quality (
internal/backtest)BrierScore: mean squared error of top-move confidence vs. realized outcome (label = positive forward return of the top thesis, falling back to the analyst kill label).ForwardReturn: top thesis return from decision time to scenario end. Computed after the replay — no future data reaches a decision.NoiseRobustness: share of non-kill events that did not trigger a material replan.These make later scoring changes (winProb priors from fundamentals, vol scaling, calibration) quantifiable against the shipped scenario.
On the shipped scenario
The scenario's "max 10% drawdown" constraint now shows up in the rationale:
...; risk budget per goal constraints: drawdown_limit 10%.Testing
EffectiveRiskBudget/parsePercent; planner tests for constraint-shrunk sizing, neutral flat-prior EV, and ticker-targeted hints; harness tests for the new metrics.go build ./... && go test ./... -racepasses;gofmt/go vetclean.