Investing decision quality: prior, time-scale coherence, calibration, portfolio risk#11
Merged
Merged
Conversation
DDE_FINANCE_HYBRID was parsed but never consumed: newEngine never set PlannerDeps.FinanceInner, and the finance route bypassed GuidedPlanner, so the investing pack's prompt template never reached the narrator. The flag now wires the investing domain's text planner as the inner narrator. The finance planner passes the pack template as the narrator's system prompt override and includes the authoritative numeric scores in its note, so the narrative can reference the numbers it annotates. The narration only ever lands in the reasoning summary; provenance records the narrator as a contributor. Numeric output is unchanged.
The Brier score averaged over two decisions in one scenario; that is too thin a base to justify scoring changes. Add three fixture scenarios: - noise gauntlet: macro/price/earnings chatter that must not replan, anchored by one real thesis break (robustness 1.0, recall 1.0) - rotation: an ACME thesis break must rotate the top move onto GLOBEX - slow drawdown: a position bleeding without a headline break; recall is 0 and Brier 0.31 because the price-move mean-reversion tilt never cuts. The scenario exists to measure progress on that blind spot. RunMatrix replays every scenario under every named scoring config (as a policy override, exactly like DDE_POLICY) in deterministic order, with RenderMatrix emitting a comparison table.
The win probability was a flat 0.5 unless a signal carried a hint, so initial plans ranked theses with no information content even though the provider serves point-in-time fundamentals that were never read. WinProbPrior tilts the 0.5 base rate by small, transparent amounts: valuation (PE bands, neutral 16-22), price/book extremes, earnings quality (EPS sign) and trailing momentum beyond +-10%, clamped to [0.35, 0.65] and documented as a heuristic, not a calibrated forecast. The planner feeds it from the fundamentals and bars it already fetches; a signal hint still wins outright for the ticker it names, since it carries event information the slow-moving prior cannot. On the embedded fixtures ACME (PE 14, positive EPS) now scores p=0.54 and GLOBEX (PE 30, PB 5) p=0.47 on otherwise uninformed plans. The move rationale exposes the probability as "(p=N.NN)". Corpus Brier is unchanged within noise: signal-driven decisions are untouched, and the terminal decisions of each scenario have degenerate (zero-length) forward windows - extending the corpus evaluation horizon is future harness work.
Three scale problems made the numeric inputs near-constant: - Volatility was per-bar (sample stddev of whatever cadence the bars have), so the 5% stop floor dominated almost always. The stop is now the per-bar vol projected onto the holding window (goal horizon, capped at 90 days) by sqrt-time, with the bar cadence inferred from the series dates. Floor 5%, cap 35%. - The upside was always RewardRiskRatio * stop. A targeted valuation signal now sets the upside to the actual gap to fair value, so a 30% revaluation is materially different from the default 2:1 assumption (the lifecycle scenario now correctly versions the plan on its valuation event; lifecycle expectations re-baselined). - HorizonFit compared the goal horizon against a price-move signal's lookback window - dead weight at 0.5 on every initial plan. It now compares the goal horizon against the vol-implied time for the thesis to traverse its upside (random-walk crossing-time heuristic), neutral when either is unknown. On the fixtures a 1-year goal fits ACME's implied ~1-year move at 0.99; a 30-day goal scores 0.25. The slow-drawdown blind spot persists (recall 0): vol expansion alone does not cross the materiality threshold while the price-move hint tilts toward mean reversion. That remains the calibration target.
Outcomes were recorded against immutable plan versions but nothing ever read them back. This adds the full loop, with the scoring math untouched: - finance.FitCalibration bins (stated confidence, realized outcome) samples and pools adjacent violators (weighted PAVA) into a monotone curve; Apply interpolates between bin midpoints. An empty curve is the identity, so absence never changes behaviour. - Service.CalibrationSamples walks a domain's goals and pairs each decisive outcome (success/failure) with the confidence the immutable plan version stated for that move. Partial/inconclusive outcomes are skipped. - `dde calibrate` fits a curve from the store and prints the DDE_POLICY snippet that installs it; the policy carries the curve per domain and the wire layer threads it into the finance planner, which applies it to stated confidence. - backtest.WalkForward fits on the first N corpus decisions and scores the rest raw vs calibrated - out-of-sample evidence for whether a curve should ship. On the current 12-decision corpus it says no (calibrated 0.327 vs raw 0.286), which is the gate doing its job; the test pins that verdict and flags when corpus growth flips it. DomainPolicy yaml tags gain omitempty so emitted snippets stay clean.
Per-thesis sizing was independent: five theses each at the 2% per-trade cap could stack 10% of equity at risk with no governor. finance.AggregateRisk computes the portfolio-level capital at risk, sqrt(r' P r) with r_i = fraction * stop distance and P the pairwise return correlations (Pearson over the most recent overlapping bars; unknown or flat series conservatively assume full co-movement). RiskBudget gains MaxAggregateRiskPct (default 6%, zero disables); when the cap binds, the planner scales every suggested size by a uniform factor - risk is linear in a uniform scale - and names the cap and the factor in each thesis rationale. A broken thesis holds no position and consumes no budget. The default cap does not bind on the embedded fixtures, so corpus metrics are unchanged.
Review of this branch surfaced ten defects; this addresses all of them. Portfolio cap (the worst): when the aggregate cap bound, only a text note was appended - SuggestedFraction and the stated "suggested size N% of equity" stayed unscaled, so consumers following the numbers carried the over-stacked risk the cap exists to prevent. Scoring and rendering are now split (scoreThesis computes, buildMove renders after the portfolio post-pass), so the stated size IS the scaled one, with "capped by portfolio_risk" and the scale factor named in the rationale. EV/prior regression: WinProbPrior reported informed=true for any ticker with two returns even when fundamentals were absent and the momentum tilt was zero, re-enabling the volatility-rewarding EV path under an effectively flat prior. A zero net tilt now reads as uninformed. Calibration loop: stored confidences were post-calibration, so a refit trained on curve-A outputs but installed over raw - diverging instead of converging. Moves now carry RawConfidence; CalibrationSamples and the walk-forward eval fit and evaluate against it, matching the domain a fresh curve is applied to. Ranking ties (a sparse curve collapses distinct composites) break on raw confidence, never input order. Config safety: ScoringConfig.Normalize fills zero fields of a partial override with defaults, so setting one knob (e.g. only max_aggregate_risk_pct) no longer zeroes KellyFraction or silently drops the aggregate cap for pre-existing policy overrides; negative cap values are the explicit disable. A drawdown_limit constraint now tightens MaxAggregateRiskPct too, so the book-level budget cannot exceed the goal's stated tolerance. docs/domains.md updated. Correlation: return series are aligned by bar date, not trailing index, so mixed cadences or gappy histories can no longer earn a phantom hedging credit from time-misaligned pairs; unmeasurable overlap still conservatively assumes full co-movement. Valuation signals: only gaps >= 5% replace the thesis upside; a near-zero gap previously had its manufactured 0.05 floor slash EV and could version the plan on noise ticks. Backtest labels: a scenario's final decision has a zero-length forward window and was unconditionally labeled a failure; it now falls back to the analyst kill label. Corpus metrics are unchanged (every shipped terminal event is a kill).
Owner
Author
|
Pushed 381c94c addressing all ten review findings:
Full suite green with -race; corpus metrics unchanged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the investing quality roadmap follow-ups to #10, one commit per phase.
Phase 1 — Hybrid narration wiring (bug fix)
DDE_FINANCE_HYBRIDwas parsed but never consumed. It now wires the investing domain's text planner as the inner narrator; the narrator receives the pack's prompt template as its system prompt override plus the authoritative numeric scores in its note. Narration only lands in the reasoning summary; provenance records the narrator as a contributor. Numeric output unchanged.Phase 2 — Scenario corpus + RunMatrix
Three new fixture scenarios join the lifecycle one: a noise gauntlet (macro/price/earnings chatter that must not replan, robustness 1.0), a rotation (an ACME thesis break must put GLOBEX on top), and a slow drawdown that pins the engine's known blind spot (recall 0, Brier 0.318 — the price-move mean-reversion tilt never cuts a bleeding position).
RunMatrixreplays every scenario under every named scoring config deterministically;RenderMatrixprints the comparison table.Phase 3 — Win-probability prior from fundamentals/momentum
provider.Fundamentals()existed but was never called; winProb was a flat 0.5 absent a signal.WinProbPriornow tilts the base rate by transparent amounts (PE bands, P/B extremes, EPS sign, ±10% trailing momentum), clamped to [0.35, 0.65]. On fixtures: ACME p=0.54, GLOBEX p=0.47 on uninformed plans; rationales expose(p=N.NN). A signal hint wins outright for the ticker it names — averaging it with the prior was tried and rejected (it dampened real event information and worsened Brier).Phase 4 — Stop, upside and horizon-fit scale with real time
Phase 5 — Outcome-driven calibration
finance.FitCalibration: binned (confidence, outcome) samples pooled by weighted PAVA into a monotone curve;Applyinterpolates; empty curve is identity.Service.CalibrationSamplespairs each decisive outcome with the confidence its immutable plan version stated; partial/inconclusive skipped.dde calibratefits from the store and prints theDDE_POLICYsnippet; policy carries the curve per domain and the wire layer threads it into the planner.backtest.WalkForwardis the out-of-sample gate: on the current 12-decision corpus a fitted curve makes held-out Brier worse (0.327 vs 0.286), so no default curve ships — the test pins that verdict and flags when corpus growth flips it.Phase 6 — Portfolio-level risk aggregation
Per-thesis sizing could stack unbounded correlated risk (five theses × 2% = 10%).
AggregateRiskcomputes sqrt(r'Pr) with Pearson correlations over overlapping bars (unknown/flat series conservatively assume full co-movement). NewRiskBudget.MaxAggregateRiskPct(default 6%, 0 disables); when it binds, all sizes scale by a uniform factor named in each rationale. Broken theses consume no budget.Corpus metrics after this PR
¹ the valuation-gap event is now correctly material; the metric only credits kill reactions. The slow-drawdown row is the deliberate blind-spot benchmark for future work.
Testing
go build ./... && go test ./... -racegreen;gofmt/go vetclean; everything offline-deterministic. New table-driven tests acrossfinance(prior, timescale, calibrate, portfolio, constraints), planner behavior (hybrid, calibration, portfolio cap, horizon, hint targeting), backtest (corpus, matrix, walk-forward) and service calibration sampling.