CPU and I/O latency reach robust-baseline parity in Darling (#1743 follow-up)

CHANGELOG.md

@@ -79,6 +79,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- **CPU and I/O latency reach robust-baseline parity in Darling** ([#1743] follow-up) - phase 1 shipped with a stated asymmetry: Darling's CPU and I/O baselines read sum/sumsq rollups that can reconstruct mean and stddev but structurally cannot produce a median, so those two families degraded to the classical gate while Lite (which reads raw grain locally) got robust statistics on all nine non-event metrics. The obvious fix was wrong twice before it was right, and both wrongs were caught by measurement rather than shipped: a 4-day raw window would satisfy sample counts but never the distinct-day trust floors (measured on the production store: zero of 5,044 full buckets trustworthy), which would have REGRESSED those families to the absolute-fallback bar - and the "4 days of raw supply" premise itself turned out not to apply here, because cpu_utilization and file_io_stats carry their own 30-DAY service-side retention (1-minute cadence collectors; verified against the production store, where both tables hold the store's full life, compressed after a day, with no TimescaleDB retention policy). So both arms now read their raw hypertables at Lite's exact grain through the same robust scaffold - the mean/stddev they produce are the SAME per-sample statistics the rollups reconstructed, plus the median/MAD the rollups could not - validated on the production fleet's busiest tenant at 227 ms for the full window, with the tier ladder behaving correctly at every store age (42 full buckets already trustworthy at 16 days, every hour-only sentinel trustworthy). The supply pins moved with the reasons: the nine aggregate-fed families keep their pin, the two raw reads are pinned to their exact tables, and a NEW pin makes the load-bearing invariant a red build instead of a quiet degradation - if either collector's retention ever drops below the 30-day baseline window, the build fails naming #1757. The retired rollup aggregates stay registered for upgrade compatibility; removing them is separate cleanup.
+
 - **The anomaly engine judges deviations against median/MAD instead of mean/stddev, and its confidence is honest** ([#1743] phase 1) - the classical baseline is self-poisoning: every burst a server ever ran through inflates its mean and stddev, and the inflation MASKS the next real deviation. Measured twice over before a line changed. On a HammerDB store, weeks of load tests left the mean 17x the median and the stddev at 11,994 against a MAD of 11 - a genuine 26x workload surge registered 0.0 classical sigmas (the engine ran and could not see it) while the modified z-score read 99.2. And on 52 production replicas, the realistic form: the fleet's busiest tenant's own history inflated its stddev enough that a real sustained 2-3x Friday-evening surge read 1.4-2.0 classical sigmas - invisible at any sane threshold - while the modified-z read 3.5-4.7 and fired. At the SAME 3.5 cutoff over 24 hours of fleet samples, the robust statistic traded +284 genuine catches for 7 misses. Both apps' baseline providers now compute median and MAD alongside mean and stddev, EXACTLY at every tier - medians cannot be pooled from per-bucket medians, so the hour-only and flat tiers come from the SQL itself (GROUPING SETS; DuckDB's native `median()`/`mad()` in Lite, `percentile_cont` twice in Darling's Postgres, validated to-the-digit against the production store's independently-computed numbers) - and the shared gate judges the modified z-score at 3.5, or 5.0 for the heavy-tailed families (waits, query duration), where the fleet sweep showed 3.5 runs hot. The wait-profile detector's ratio trigger is REPLACED by the modified-z: measured at every cutoff swept, the ratio caught nothing the modified-z missed (strict containment) while missing the masked-surge class entirely - and the scorer grades those facts off the same statistic, because a catch the ratio floor zeroes at scoring is not a catch. Everything defensive is unchanged and load-bearing: the magnitude floors and absolute-fallback bars apply exactly as before (fleet-measured, MAD collapsed ONLY on idle-box CPU, precisely where the bounded-metric floor clamps), the trust gate and fallback complementarity are untouched, and a bucket without robust statistics - Darling's CPU and I/O latency read sum/sumsq rollups that cannot produce a median (their raw-window variants are follow-up work; Lite reads raw grain locally and gets robust statistics on all nine non-event metrics) - degrades to the classical gate rather than judging against zeroed fields. Fact confidence is now derived from baseline quality (tier and sample density; an untrustworthy baseline scores zero) instead of the hardcoded 1.0 shipped since [#1606], and every anomaly fact carries its baseline median and MAD so a reader can see the frame the verdict was made in. The calibration datasets ride in the regression suite verbatim - the HammerDB blindness case, the production surge, the floor-composition cases - so the thresholds stay measurements, not folklore. EWMA and changepoint detection remain phased behind this, per the issue.
 
 - **The MCP docs now lead with the boundary instead of the warning** - the networked-MCP section of the Darling guide opened with everything a token-holder could do, under a "blast radius" heading, and read like an AI client had the run of your SQL Servers. That was never what the code does: no MCP tool runs SQL an AI client wrote against a monitored server, the only live-server contact (the analysis plan fetch and the onboarding connection probe) runs the product's own fixed read-only queries under the same least-privilege monitoring login the collectors use, and every write-capable tool changes the monitor's own configuration under the carved-down `mcp` store role. The facts and every piece of wire/TLS guidance are unchanged - the sections now state what is structurally impossible first, then what the token actually gates, and the root README's MCP section carries the same boundary up front.

Darling/Darling.Tests/BaselineSupplyTests.cs

@@ -11,6 +11,7 @@
 using System.Linq;
 using System.Runtime.CompilerServices;
 using PerformanceMonitor.Analysis.Baselines;
+using PerformanceMonitor.Collectors;
 using PerformanceMonitor.Darling.Analysis;
 using PerformanceMonitor.Darling.Storage;
 using Xunit;
@@ -134,27 +135,52 @@ public void NoBaselineFamily_StillReadsRawHistory()
     }
 
     /// <summary>
-    /// The eleven families are served by the nine aggregates and nothing else — a family pointed at a view
+    /// Nine families are served by the baseline aggregates and nothing else — a family pointed at a view
     /// that no longer exists would fail at runtime against a real store, which no unit test would catch.
+    /// <para>#1743 follow-up: CPU and I/O latency are the two deliberate EXCEPTIONS — they read their RAW
+    /// hypertables (at Lite's grain, so the robust scaffold applies; their retired sum/sumsq rollups could
+    /// not produce a median). That is safe from #1757 only because those two collectors carry their own
+    /// 30-day service-side retention, which <see cref="RawBaselineFamilies_RetentionCoversTheWindow"/>
+    /// pins as the load-bearing invariant.</para>
     /// </summary>
     [Fact]
-    public void EveryFamily_ReadsOneOfTheNineBaselineAggregates()
+    public void EveryFamily_ReadsItsIntendedSupply()
     {
         var views = TimescaleSupport.BaselineAggregates.Select(a => a.View).ToArray();
-        var families = new[]
+        var aggregateFamilies = new[]
         {
-            MetricNames.Cpu, MetricNames.BatchRequests, MetricNames.WaitStats, MetricNames.SessionCount,
-            MetricNames.QueryDuration, MetricNames.IoLatency, MetricNames.Blocking, MetricNames.Deadlock,
+            MetricNames.BatchRequests, MetricNames.WaitStats, MetricNames.SessionCount,
+            MetricNames.QueryDuration, MetricNames.Blocking, MetricNames.Deadlock,
             MetricNames.Memory, MetricNames.WaitMsPerSec, MetricNames.BlockingPerMinute,
         };
 
-        foreach (var family in families)
+        foreach (var family in aggregateFamilies)
         {
             var sql = PgBaselineProvider.GetBaselineQuery(family)!;
             Assert.True(
                 views.Any(v => sql.Contains("FROM " + v, StringComparison.Ordinal)),
                 $"{family} does not read any known baseline aggregate");
         }
+
+        Assert.Contains("FROM cpu_utilization_stats", PgBaselineProvider.GetBaselineQuery(MetricNames.Cpu)!, StringComparison.Ordinal);
+        Assert.Contains("FROM file_io_stats", PgBaselineProvider.GetBaselineQuery(MetricNames.IoLatency)!, StringComparison.Ordinal);
+    }
+
+    /// <summary>
+    /// The invariant that makes the CPU/I-O raw reads safe: both collectors retain at least the full
+    /// baseline window. Drop either below <see cref="BaselineMath.BaselineWindowDays"/> and the family
+    /// silently regresses to #1757's short-supply shape — this pin is what makes that a red build
+    /// instead of a quiet baseline degradation.
+    /// </summary>
+    [Fact]
+    public void RawBaselineFamilies_RetentionCoversTheWindow()
+    {
+        Assert.True(
+            CollectorScheduleDefaults.All["cpu_utilization"].RetentionDays >= BaselineMath.BaselineWindowDays,
+            "cpu_utilization retention no longer covers the baseline window");
+        Assert.True(
+            CollectorScheduleDefaults.All["file_io_stats"].RetentionDays >= BaselineMath.BaselineWindowDays,
+            "file_io_stats retention no longer covers the baseline window");
     }
 
     /// <summary>

Darling/Darling.Tests/DarlingAnomalyBaselineTests.cs

@@ -112,13 +112,19 @@ all the way out or the same overflow returns by a different route. */
         Assert.Contains("delta_stall_read_ms::DOUBLE PRECISION", TimescaleSupport.CreateFileIoBaselineSql, StringComparison.Ordinal);
         Assert.DoesNotContain("delta_stall_read_ms * 1.0", TimescaleSupport.CreateFileIoBaselineSql, StringComparison.Ordinal);
 
+        /* #1743 follow-up moved the arm off the rollup and onto the raw hypertable (the rollup
+           cannot produce a median) — the cast pin moves WITH it: the ratio must still be computed
+           in float arithmetic at the source. The old SUM(row_count) pin's SEMANTIC survives as the
+           nullable-v design: the arm's WHERE keeps write-only rows (delta_reads > 0 OR
+           delta_writes > 0) and must NOT filter the NULL ratios out — the scaffold's COUNT(*)
+           counts them (the row_count behavior) while AVG/STDDEV/median/mad ignore them (the
+           ratio_count behavior), exactly the retired rollup's two-count distinction. */
         var sql = PgBaselineProvider.GetBaselineQuery(MetricNames.IoLatency)!;
-        Assert.Contains("file_io_baseline", sql, StringComparison.Ordinal);
-        Assert.Contains("SQRT(", sql, StringComparison.Ordinal);
-        /* sample_count must come from row_count, NOT ratio_count: the raw path counted rows whose ratio was
-           NULL (writes but no reads pass the filter and average to nothing), so counting only the non-null
-           ratios would silently under-report the sample size the baseline gate reads. */
-        Assert.Contains("SUM(row_count) AS sample_count", sql, StringComparison.Ordinal);
+        Assert.Contains("FROM file_io_stats", sql, StringComparison.Ordinal);
+        Assert.Contains("delta_stall_read_ms::DOUBLE PRECISION / NULLIF(delta_reads, 0)", sql, StringComparison.Ordinal);
+        Assert.DoesNotContain("delta_stall_read_ms * 1.0", sql, StringComparison.Ordinal);
+        Assert.Contains("(delta_reads > 0 OR delta_writes > 0)", sql, StringComparison.Ordinal);
+        Assert.DoesNotContain("v IS NOT NULL", sql, StringComparison.Ordinal);
     }
 
     /* ---------------- ungated: method-surface pins vs Lite ---------------- */

Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs

@@ -145,9 +145,9 @@ public void InvalidateCache(int serverId)
 
             using var reader = await cmd.ExecuteReaderAsync();
             /* #1743: the robust-scaffold metrics return eight columns (…, median_val, mad_val)
-               and carry sentinel tier rows; the two rollup-bound metrics (CPU, I/O) still return
-               the six-column classical shape — detected by column count, so their buckets read
-               Median=0/Mad=0 and the robust path degrades to the classical one for them. */
+               and carry sentinel tier rows; the two event-family metrics (blocking, deadlock)
+               keep the six-column classical shape — detected by column count, so their buckets
+               read Median=0/Mad=0 and the robust path degrades for them. */
             var hasRobustColumns = reader.FieldCount >= 8;
             while (await reader.ReadAsync())
             {
@@ -241,12 +241,13 @@ JOIN tier_mads AS m
     /// The eleven per-metric baseline queries — Lite's, verbatim, except the four QUALIFY
     /// sites rewritten for Postgres (no QUALIFY support). Internal (not private like Lite's)
     /// so Darling.Tests can pin every query's dialect and the rewrites' structure ungated.
-    /// <para>#1743: the seven raw-grain metrics route their cleaned rowsets through
-    /// <see cref="RobustTierScaffold"/> and return EIGHT columns (…, median_val, mad_val).
-    /// CPU and I/O latency read pre-aggregated sum/sumsq rollups that cannot produce a median —
-    /// they keep the six-column classical shape until their raw-window variants land, and the
-    /// reader detects the shape by column count. Blocking/deadlock are event-family (events/day,
-    /// stddev 0) evaluated on the event-ratio path, deliberately untouched.</para>
+    /// <para>#1743: the nine non-event metrics route their cleaned rowsets through
+    /// <see cref="RobustTierScaffold"/> and return EIGHT columns (…, median_val, mad_val) — CPU
+    /// and I/O latency included, reading their RAW hypertables at Lite's grain (their retired
+    /// sum/sumsq rollups could not produce a median; both tables carry their own 30-day
+    /// service-side retention, so this does not reopen #1757 — see the arms' notes).
+    /// Blocking/deadlock are event-family (events/day, stddev 0) evaluated on the event-ratio
+    /// path, deliberately untouched; the reader detects their six-column shape by count.</para>
     /// </summary>
     internal static string? GetBaselineQuery(string metricName)
     {
@@ -258,19 +259,22 @@ JOIN tier_mads AS m
         // collection_time first, then bucket by hour+dow.
         return metricName switch
         {
-            // Point-in-time metric — no restart exclusion needed
+            /* #1743 follow-up: CPU reads the RAW hypertable, at Lite's exact per-sample grain, so
+               the robust scaffold applies — the old sum/sumsq rollup could reconstruct mean/stddev
+               but structurally cannot produce a median. Reading raw here does NOT reopen #1757:
+               that finding was 4 days of supply under a 30-day window, and cpu_utilization carries
+               its own 30-DAY service-side retention (CollectorScheduleDefaults: 1-minute cadence,
+               30-day retention; verified on a production store — no TimescaleDB retention policy
+               on the table, service-side purge at 30d, compressed after 1 day). The mean/stddev
+               this computes are the SAME per-sample statistics the rollup reconstruction produced.
+               The now-unused cpu_utilization_baseline aggregate remains registered for upgrade
+               compatibility; retiring it is separate cleanup. */
             MetricNames.Cpu => @"
-SELECT EXTRACT(HOUR FROM collection_time)::INT AS hour_of_day,
-       EXTRACT(DOW FROM collection_time)::INT AS day_of_week,
-       SUM(cpu_sum) / NULLIF(SUM(cpu_count), 0) AS mean_val,
-       SQRT(GREATEST(
-           (SUM(cpu_sumsq) - POWER(SUM(cpu_sum), 2) / NULLIF(SUM(cpu_count), 0))
-           / NULLIF(SUM(cpu_count) - 1, 0), 0)) AS stddev_val,
-       SUM(cpu_count) AS sample_count,
-       COUNT(DISTINCT collection_time::DATE) AS distinct_days
-FROM cpu_utilization_baseline
-WHERE server_id = $1 AND collection_time >= $2 AND collection_time < $3
-GROUP BY hour_of_day, day_of_week",
+WITH clean AS (
+    SELECT collection_time, sqlserver_cpu_utilization::DOUBLE PRECISION AS v
+    FROM cpu_utilization_stats
+    WHERE server_id = $1 AND collection_time >= $2 AND collection_time < $3
+)," + RobustTierScaffold,
 
             /* QUALIFY rewrite 1 of 4 — cumulative counter, restart exclusion.
                Excludes samples where the delta drops to 0 when the prior sample was > 1000
@@ -399,22 +403,21 @@ FROM with_lag
     WHERE NOT (total_elapsed = 0 AND prior_total_elapsed > 100000)
 )," + RobustTierScaffold,
 
-            // Point-in-time metric — no restart exclusion needed. The stall/reads ratio is cast to
-            // DOUBLE PRECISION (as the memory / wait-rate metrics are) so a spurious large delta can't
-            // make STDDEV_SAMP produce a numeric that overflows System.Decimal when Npgsql materializes
-            // the aggregate (it does with `* 1.0`, which yields numeric, not float8).
+            /* #1743 follow-up: same move as CPU — raw hypertable at Lite's per-file-row grain so
+               the robust scaffold applies (file_io_stats also carries its own 30-day service-side
+               retention; see the CPU arm's note). The stall/reads ratio keeps its DOUBLE PRECISION
+               cast so a spurious large delta can't make STDDEV_SAMP produce a numeric that
+               overflows System.Decimal when Npgsql materializes the aggregate. v stays NULLABLE
+               (a write-only file row has no read latency): AVG/STDDEV/median/mad all ignore those
+               rows while COUNT(*) keeps counting them — exactly the row_count-vs-ratio_count
+               distinction the retired rollup documented, preserved at the raw grain. */
             MetricNames.IoLatency => @"
-SELECT EXTRACT(HOUR FROM collection_time)::INT AS hour_of_day,
-       EXTRACT(DOW FROM collection_time)::INT AS day_of_week,
-       SUM(ratio_sum) / NULLIF(SUM(ratio_count), 0) AS mean_val,
-       SQRT(GREATEST(
-           (SUM(ratio_sumsq) - POWER(SUM(ratio_sum), 2) / NULLIF(SUM(ratio_count), 0))
-           / NULLIF(SUM(ratio_count) - 1, 0), 0)) AS stddev_val,
-       SUM(row_count) AS sample_count,
-       COUNT(DISTINCT collection_time::DATE) AS distinct_days
-FROM file_io_baseline
-WHERE server_id = $1 AND collection_time >= $2 AND collection_time < $3
-GROUP BY hour_of_day, day_of_week",
+WITH clean AS (
+    SELECT collection_time, delta_stall_read_ms::DOUBLE PRECISION / NULLIF(delta_reads, 0) AS v
+    FROM file_io_stats
+    WHERE server_id = $1 AND collection_time >= $2 AND collection_time < $3
+    AND   (delta_reads > 0 OR delta_writes > 0)
+)," + RobustTierScaffold,
 
             // Event-based — mean = events per day for this bucket, sample_count = distinct days observed.
             // No restart exclusion needed (event counts, not cumulative).