Skip to content

Improve investing decision quality: constraints, EV neutrality, eval metrics#10

Merged
vingrad merged 1 commit into
mainfrom
feat/investing-decision-quality
Jun 10, 2026
Merged

Improve investing decision quality: constraints, EV neutrality, eval metrics#10
vingrad merged 1 commit into
mainfrom
feat/investing-decision-quality

Conversation

@vingrad

@vingrad vingrad commented Jun 10, 2026

Copy link
Copy Markdown
Owner

Quick-wins pass on decision quality for the investing pack, found while evaluating the finance planner's scoring path.

What changed

Per-goal risk constraints now bind sizing (internal/finance/constraints.go, new)
The investing pack validates drawdown_limit / risk_tolerance constraints and even warns "sizing will fall back to conservative defaults" when they're missing — but they were never read; the risk budget came solely from pack/policy config. EffectiveRiskBudget now tightens the budget per goal:

  • a drawdown limit caps MaxPositionPct (a total loss of one position can't exceed the stated limit) and MaxPortfolioRiskPct (five consecutive stop-outs stay within it),
  • conservative tolerance halves the Kelly fraction and concentration cap; aggressive scales the Kelly fraction only (hard caps stay as guardrails),
  • unparseable constraints leave the base unchanged, and the move rationale names what bound.

EV no longer rewards volatility
With the default flat winProb=0.5 and winFrac = RewardRiskRatio * lossFrac, expected value reduced to 0.5*lossFrac*(ratio-1) — monotonically increasing in volatility, so every initial plan tilted toward the most volatile ticker. The EV component is now neutral (NeutralEVScore = 0.5) until a signal carries an actual probability hint. Relatedly, a signal hint now only informs the ticker it names (a valuation hint on ACME no longer lifted GLOBEX too); unnamed signals (e.g. macro) still apply to all candidates.

Backtest harness measures decision quality (internal/backtest)

  • BrierScore: mean squared error of top-move confidence vs. realized outcome (label = positive forward return of the top thesis, falling back to the analyst kill label).
  • Per-decision ForwardReturn: top thesis return from decision time to scenario end. Computed after the replay — no future data reaches a decision.
  • NoiseRobustness: share of non-kill events that did not trigger a material replan.

These make later scoring changes (winProb priors from fundamentals, vol scaling, calibration) quantifiable against the shipped scenario.

On the shipped scenario

Backtest: ACME thesis lifecycle
  versions created : 1
  kill precision   : 1.00
  kill recall      : 1.00
  brier score      : 0.054 (lower is better; 0.25 == coin-flip confidence)
  noise robustness : 1.00 (share of non-kill events not reacted to)
  hypothetical pnl : +8.00% (illustrative, not a strategy return)

The scenario's "max 10% drawdown" constraint now shows up in the rationale: ...; risk budget per goal constraints: drawdown_limit 10%.

Testing

  • Table-driven tests for EffectiveRiskBudget/parsePercent; planner tests for constraint-shrunk sizing, neutral flat-prior EV, and ticker-targeted hints; harness tests for the new metrics.
  • go build ./... && go test ./... -race passes; gofmt/go vet clean.
  • No wiring, policy, schema, or pack-descriptor changes; everything stays offline-deterministic.

…metrics

Three fixes to the finance planner and backtest harness:

- Honor per-goal risk constraints in sizing. drawdown_limit and
  risk_tolerance constraints were validated and warned about but never
  read; the risk budget came only from pack/policy config. A drawdown
  limit now caps the concentration and per-trade risk budgets, and a
  conservative/aggressive tolerance scales the Kelly fraction. The move
  rationale names what bound.

- Stop rewarding volatility in the EV score. With a flat 0.5 win
  probability and winFrac = ratio*lossFrac, expected value grew
  monotonically with volatility, so the EV component favoured the most
  volatile ticker on every initial plan. The EV score is now neutral
  (0.5) until a signal tilts the odds, and a signal hint only informs
  the ticker it names.

- Add decision-quality metrics to the backtest report: a Brier score of
  top-move confidence against realized forward returns, per-decision
  forward-return attribution, and a noise-robustness rate (share of
  non-kill events that did not trigger a replan). Forward returns are
  computed after the replay, so no future data reaches a decision.
@vingrad vingrad merged commit bbaeaa5 into main Jun 10, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant