From 330f32bca7867520a27d57991d7b353e617a1ca8 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 19 Jun 2026 15:44:11 +0000 Subject: [PATCH] perturbation-sim: CHAODA epicenter probe (negative) + IT cross-grid replication of family-basin Weyl MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two results closing the three-axis surge synthesis on real PyPSA data. (1) CHAODA epicenter — NEGATIVE, refutes my own conjecture. New gated example chaoda_surge_epicenter.rs (--features ndarray-simd) uses the REAL ndarray::hpc::clam ClamTree + anomaly_scores. ES, 64-bit Fiedler-sign fingerprints: the brittle 2-line seam (4 nodes) is LESS anomalous than average (0.302 vs 0.339, ratio 0.89; top-quartile lift 1.00 = chance). CHAODA's LFD anomaly does NOT single out the spectral-cut bottleneck — Fiedler-sign fingerprints put boundary nodes in a dense low-LFD region. The "WHERE = CHAODA anomaly" synthesis claim is WITHDRAWN (one encoding, not tuned to win). Epiphany E-CHAODA-IS-NOT-THE-SEAM-EPICENTER. (2) IT replication — CONFIRMS E-FAMILY-BASIN-WEYL across grids. The family_basin_weyl_multihop probe on IT (192 buses, C=2.38): HEEL 1.30 + HIP 1.35 HOP-LOCAL, LEAF leaks — same crisp-tier-localizes / fine-tier-leaks shape as ES, on an independent grid. Not an ES artifact. Epiphany E-FAMILY-BASIN-WEYL-IT-REPLICATION. CHAODA example gated behind ndarray-simd (real ndarray dep, git form); default build compiles the stub. Verified locally against the ndarray sibling (the committed dep stays the git fork per P0). Cargo.lock unchanged (path resolution not committed). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi --- .claude/board/EPIPHANIES.md | 29 ++++ crates/perturbation-sim/Cargo.toml | 4 + .../examples/chaoda_surge_epicenter.rs | 159 ++++++++++++++++++ 3 files changed, 192 insertions(+) create mode 100644 crates/perturbation-sim/examples/chaoda_surge_epicenter.rs diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 819a4358..3075754d 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,32 @@ +## 2026-06-19 — E-CHAODA-IS-NOT-THE-SEAM-EPICENTER — REFUTES this session's own "WHERE = CHAODA anomaly" conjecture: on the real ES grid with a Fiedler-sign fingerprint, the brittle 2-line seam is NOT a CHAODA/LFD anomaly (it is LESS anomalous than average); the geometric outlier and the spectral-cut bottleneck do not coincide + +**Status:** FINDING (measured NEGATIVE; probe `perturbation-sim/examples/chaoda_surge_epicenter.rs`, gated `--features ndarray-simd`, real `ndarray::hpc::clam::{ClamTree, anomaly_scores}`). **Refutes** the conjecture I posted earlier this session ("a surge's epicenter CAN be modeled as a CHAODA anomaly — the WHERE axis"). The user asked to model CLAM/CHAODA with the real ndarray engine; I did, and it killed the conjecture — measured, not asserted. + +**What the probe measured (ES largest component, 261 buses):** each node → a 64-bit Fiedler-sign fingerprint (the spectral embedding) → real ndarray `ClamTree::build` (Hamming) → `anomaly_scores` (leaf-cluster LFD, normalized). The HHTL seam = the inter-HEEL-basin cut (4 nodes — exactly the 2-line bottleneck). +- mean anomaly, **SEAM** nodes: **0.302** — mean anomaly, ALL nodes: **0.339** → ratio **0.89** (seam is *less* anomalous); +- seam share of the top-quartile anomalies: 1/4, **lift 1.00** (pure chance). +**VERDICT: NOT SUPPORTED** (gate was ratio ≥ 1.30 AND lift ≥ 1.30). + +**Why (clear in hindsight):** Fiedler-sign fingerprints cluster nodes *by basin*, so boundary/cut nodes sit in a **dense, low-LFD** region → low anomaly. CHAODA flags **high-LFD** geometric complexity, which lives elsewhere (not at the spectral cut). The brittle cut is not a geometric outlier on this embedding. So **CHAODA ≠ epicenter detector** as I'd claimed; the three-axis "surge" decomposition's WHERE leg is **not** simply a CHAODA anomaly — that synthesis claim was over-reach, now corrected. + +**Honest scope / non-tuning:** this is ONE encoding (Fiedler-sign) on ONE grid; I did NOT keep swapping encodings until it went green (that's the confirmation-bias trap in reverse). The defensible statement: *on the natural spectral-embedding encoding, CHAODA's LFD anomaly does not identify the seam.* A different fingerprint (e.g. raw bus features, or distance-to-cut) might behave differently — but the clean "epicenter = CHAODA anomaly" claim is **withdrawn** pending such a probe. CHAODA remains a real manifold-anomaly detector; it just doesn't detect *this* (the spectral bottleneck). Cross-refs: `E-CLAM-IS-THE-MANIFOLD-ENGINE` (CHAODA = LFD repulsion — still true, just not seam-aligned), `E-FAMILY-BASIN-WEYL-HOP-LOCAL-AT-CRISP-TIER` (the HOW-it-spreads axis, which DID hold), `ndarray::hpc::clam`. + +--- + +## 2026-06-19 — E-FAMILY-BASIN-WEYL-IT-REPLICATION — the crisp-tier hop-locality reinstatement replicates on the IT grid (independent C-solid grid), strengthening `E-FAMILY-BASIN-WEYL-HOP-LOCAL-AT-CRISP-TIER` from one-grid to cross-grid + +**Status:** FINDING (confirmation; same probe `family_basin_weyl_multihop`, country=IT). Run on the IT largest component (192 buses / 259 lines) — the other C-solid grid (probe C `(λ₃−λ₂)/λ₂ = 2.38`): + +| tier | basins | within | seam | ratio | verdict | +|---|---|---|---|---|---| +| HEEL | 2 | 0.730 | 0.561 | 1.30 | **HOP-LOCAL** | +| HIP | 4 | 0.791 | 0.587 | 1.35 | **HOP-LOCAL** | +| LEAF | 8 | 0.592 | 0.512 | 1.16 | leaks | + +IT **replicates and slightly extends** the ES result: the crisp band is hop-local at HEEL **and HIP** (ES was HEEL only), LEAF leaks — same shape (crisp tiers localize, finest tier leaks), on an independent grid. So the family-basin Weyl-multi-hop hop-locality is **not an ES artifact** — it holds where the bisection is stable, across grids. The tier at which leakage sets in is grid-specific (the marginal-DK / out-of-family-tie-growth boundary), as expected. Cross-ref: `E-FAMILY-BASIN-WEYL-HOP-LOCAL-AT-CRISP-TIER` (the ES finding this confirms). + +--- + ## 2026-06-19 — E-FAMILY-BASIN-WEYL-HOP-LOCAL-AT-CRISP-TIER — REINSTATEMENT (tier-scoped) of `E-OUTAGE-CASCADE-IS-NON-LOCAL`: WITH the family-basin partition, a within-basin perturbation is block-localized (95.7 % containment, within/seam ratio 1.54) at the **crisp top split**, so multi-hop Weyl IS hop-local there; finer tiers leak. The earlier non-locality was the *un-partitioned* global solve. (Davis-Kahan is the *mechanism* — see body — NOT the numeric evidence; the gate is the containment ratio.) **Status:** FINDING (measured, real ES PyPSA data; probe `perturbation-sim/examples/family_basin_weyl_multihop.rs`). **Tier-scoped reinstatement** of the reach claim that `E-OUTAGE-CASCADE-IS-NON-LOCAL` had corrected. Operator hypothesis: "with family basin it works now with weyl multihop." Confirmed — at the crisp tier only. diff --git a/crates/perturbation-sim/Cargo.toml b/crates/perturbation-sim/Cargo.toml index 18d06c48..5e8b0b6c 100644 --- a/crates/perturbation-sim/Cargo.toml +++ b/crates/perturbation-sim/Cargo.toml @@ -124,3 +124,7 @@ path = "examples/outage_over_hhtl_hops.rs" [[example]] name = "family_basin_weyl_multihop" path = "examples/family_basin_weyl_multihop.rs" + +[[example]] +name = "chaoda_surge_epicenter" +path = "examples/chaoda_surge_epicenter.rs" diff --git a/crates/perturbation-sim/examples/chaoda_surge_epicenter.rs b/crates/perturbation-sim/examples/chaoda_surge_epicenter.rs new file mode 100644 index 00000000..c9633625 --- /dev/null +++ b/crates/perturbation-sim/examples/chaoda_surge_epicenter.rs @@ -0,0 +1,159 @@ +//! PROBE — CHAODA epicenter (`ndarray::hpc::clam`) on the real ES grid. +//! **Gated behind `ndarray-simd`** (uses ndarray's production `ClamTree` + +//! `anomaly_scores`, the real CLAM/CHAODA engine — not perturbation-sim's lite). +//! +//! The WHERE axis of the three-axis surge decomposition +//! (`E-FAMILY-BASIN-WEYL-HOP-LOCAL-AT-CRISP-TIER`): a surge's fail-first +//! compartment is the brittle seam — geometrically anomalous on the node +//! manifold. CHAODA's LFD-based anomaly should flag it **statically, with no +//! cascade simulation**. Hypothesis: the top-anomaly nodes concentrate on the +//! HHTL seam — the inter-HEEL-basin cut, the crisp 2-split bottleneck (the +//! "without family nodes" table's HIP 2-line seam, lines 46/150). +//! +//! Encoding: each node → a binary fingerprint = the **sign of its top-K Fiedler +//! eigenvector coordinates**, packed to bytes (the spectral embedding, the same +//! Laplacian spectrum the HHTL tiers come from). `ClamTree::build` (Hamming) + +//! `anomaly_scores` → per-node LFD anomaly. We then compare the seam-cut +//! endpoints' anomaly against the global mean and their share of the top anomalies. +//! +//! Honest gate: the epicenter↔anomaly claim is SUPPORTED only if the seam nodes' +//! mean anomaly ≥ 1.30× the global mean AND seam nodes are over-represented in +//! the top-quartile anomalies (lift ≥ 1.30). Reported as-is otherwise. +//! +//! Run: cargo run --manifest-path crates/perturbation-sim/Cargo.toml \ +//! --features ndarray-simd --example chaoda_surge_epicenter \ +//! -- /tmp/pypsa/buses.csv /tmp/pypsa/lines.csv ES + +#[cfg(not(feature = "ndarray-simd"))] +fn main() { + eprintln!( + "chaoda_surge_epicenter requires `--features ndarray-simd` \ + (it uses ndarray::hpc::clam — the real CLAM/CHAODA engine)." + ); +} + +#[cfg(feature = "ndarray-simd")] +fn main() { + use ndarray::hpc::clam::ClamTree; + use perturbation_sim::{from_pypsa_csv, hhtl_keys, symmetric_eigen, Grid}; + use std::collections::HashSet; + use std::fs; + + const K: usize = 64; // spectral-embedding bits per node (Fiedler coords 1..=K) + + let args: Vec = std::env::args().collect(); + let (bpath, lpath, country) = ( + args.get(1) + .map(String::as_str) + .unwrap_or("/tmp/pypsa/buses.csv"), + args.get(2) + .map(String::as_str) + .unwrap_or("/tmp/pypsa/lines.csv"), + args.get(3).map(String::as_str).unwrap_or("ES"), + ); + let buses = fs::read_to_string(bpath).expect("read buses.csv"); + let lines = fs::read_to_string(lpath).expect("read lines.csv"); + let import = from_pypsa_csv(&buses, &lines, Some(country)) + .expect("parse pypsa") + .largest_component(); + let grid: &Grid = &import.grid; + let (n, m) = (grid.n, grid.edges.len()); + + // Spectral embedding: top-K Fiedler eigenvectors (skip j=0, the constant). + let eig = symmetric_eigen(&grid.laplacian_of(&vec![true; m]), n); + let k = K.min(n.saturating_sub(1)); + let vecs: Vec> = (1..=k).map(|j| eig.eigenvector(j)).collect(); + + // Node fingerprint = sign bits of its K Fiedler coords, packed to bytes. + let vec_len = k.div_ceil(8); + let mut data = vec![0u8; n * vec_len]; + for (node, fp) in data.chunks_mut(vec_len).enumerate() { + for (j, v) in vecs.iter().enumerate() { + if v[node] >= 0.0 { + fp[j / 8] |= 1 << (j % 8); + } + } + } + + // The real ndarray CLAM/CHAODA engine. + let tree = ClamTree::build(&data, vec_len, 4); + let scores = tree.anomaly_scores(&data, vec_len); + + // HHTL seam = the inter-HEEL-basin cut (the crisp 2-split bottleneck). + let keys = hhtl_keys(grid); + let mut seam_nodes: HashSet = HashSet::new(); + for e in &grid.edges { + if keys[e.from].heel != keys[e.to].heel { + seam_nodes.insert(e.from); + seam_nodes.insert(e.to); + } + } + + let mean = |idxs: &[usize]| -> f64 { + if idxs.is_empty() { + 0.0 + } else { + idxs.iter().map(|&i| scores[i].score).sum::() / idxs.len() as f64 + } + }; + let all: Vec = (0..n).collect(); + let seam: Vec = seam_nodes.iter().copied().collect(); + let (anom_seam, anom_all) = (mean(&seam), mean(&all)); + let anom_ratio = if anom_all > 0.0 { anom_seam / anom_all } else { 0.0 }; + + // Top-quartile anomalies: are seam nodes over-represented? + let mut ranked: Vec = (0..n).collect(); + ranked.sort_by(|&a, &b| { + scores[b] + .score + .partial_cmp(&scores[a].score) + .unwrap_or(std::cmp::Ordering::Equal) + }); + let top_n = (n / 4).max(1); + let top: HashSet = ranked[..top_n].iter().copied().collect(); + let seam_in_top = seam.iter().filter(|i| top.contains(i)).count(); + let expected = top_n as f64 * seam.len() as f64 / n as f64; // null expectation + let lift = if expected > 0.0 { + seam_in_top as f64 / expected + } else { + 0.0 + }; + + println!("PROBE — CHAODA surge epicenter (ndarray::hpc::clam, real {country} grid)"); + println!(" grid: {n} buses, {m} lines; {k}-bit Fiedler fingerprints ({vec_len} B each)"); + println!( + " seam (inter-HEEL-basin cut): {} nodes; CLAM tree built, anomaly_scores computed", + seam.len() + ); + println!("\n anomaly (LFD-derived, higher = more anomalous):"); + println!(" mean anomaly, SEAM nodes : {anom_seam:.3}"); + println!(" mean anomaly, ALL nodes : {anom_all:.3}"); + println!(" ratio (seam / all) : {anom_ratio:.2}"); + println!( + " seam share of top-{top_n} anomalies: {seam_in_top}/{} (expected {expected:.1}, lift {lift:.2})", + seam.len() + ); + + let supported = !seam.is_empty() && anom_ratio >= 1.30 && lift >= 1.30; + println!("\n VERDICT:"); + if supported { + println!( + " [SUPPORTED] the seam (the brittle fail-first cut) IS a CHAODA anomaly — its mean \ + LFD anomaly is {anom_ratio:.2}× the global mean and it is {lift:.2}× over-represented \ + in the top-quartile anomalies. The surge EPICENTER is detectable statically from the \ + manifold geometry, no cascade simulation — the WHERE axis closes on real data." + ); + } else { + println!( + " [NOT SUPPORTED] seam anomaly ratio {anom_ratio:.2} / top-quartile lift {lift:.2} \ + do not clear 1.30 — on this grid the LFD anomaly does not single out the seam. Report \ + honestly: CHAODA flags geometric outliers, which need not coincide with the \ + spectral-cut bottleneck here. Do not promote." + ); + } + println!( + " NOTE: CHAODA anomaly = leaf-cluster LFD normalized over the tree; the seam is the \ + inter-HEEL-basin cut (the crisp 2-split). This is the WHERE axis (epicenter), distinct \ + from the Weyl HOW-MUCH and the family-basin HOW-it-spreads axes." + ); +}