perf: host-batched likelihood kernel + fused --h_values grid (value-preserving, 3.8x)#31
Merged
Merged
Conversation
…nalization Physics-change protocol requirement (assert OLD numerical results before the change so the [PHYSICS] diffs are verifiable): - NEW test_bayesian_statistics_host_z_kernel.py: exact pins of single_host_likelihood (volume_deconv + local_ratio, with/without BH mass, low-z window clamp) via stubbed worker globals — the first tests to numerically execute the production host-z kernel. - test_constants.py: pin HOST_DRAW_Z_MAX == 0.5 and GALAXY_CATALOG_REDSHIFT_UPPER_LIMIT == 0.55 (pre-#20 values) + the host-draw <= population-depth ordering constraint. - physical_relations_test.py: d_L anchors at z = 0.5 / 1.5 incl. the closure-truth case dist(1.5, h=0.67) > dist(1.5, h=0.73) that the current fiducial-h parameter-space cap violates. - test_pp_coverage.py: exact-float pins of the tiny-config harness output for both kernels (the determinism test does not pin across code changes). Prep for issues #20 (depth 1.5) and #16 (sigma_v marginalization). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
User decision on issue #20 (2026-07-03): 'first go for z=1.5 and then see the results and HPC performance'. The pre-dt2 justification (horizon z ~ 0.18, truncation exact) is retired — post-dt2 the EMRI horizon reaches z ~ 1.5+, so the depth is a deliberate population-model choice matching Model1CrossCheck.max_redshift = 1.5. - constants.py: HOST_DRAW_Z_MAX 0.5 -> 1.5; GALAXY_CATALOG_REDSHIFT_UPPER_LIMIT 0.55 -> 1.55 (documented as UNWIRED: the reduced CSV is full-depth, load-time depth is max_redshift via _get_pruned_galaxy_catalog). - main.py injection_campaign: z_cut is now HOST_DRAW_Z_MAX instead of a hardcoded 0.5 — the P_det injection grid must span the host-draw volume or the selection function is blind above the cut. - cosmological_model.py: parameter-space d_L cap computed at the LOWEST campaign h (H_MIN/100 = 0.60 -> 13.0 Gpc) instead of the fiducial h = 0.73 (10.686 Gpc), which silently dropped z >~ 1.35 events in closure runs at h_true = 0.67 (dist(1.5, 0.67) = 11.643 Gpc) via ParameterOutOfBoundsError; plus an explicit ordering guard HOST_DRAW_Z_MAX <= max_redshift (the d_L pre-screen derivation relies on it). - handler.py: rewrite the stale 'truncation is exact' draw docstrings. - main.py fig23: sky-averaged completeness figure now rendered to the campaign depth. - tests: constants pins flipped 0.5->1.5 / 0.55->1.55 in the same diff (physics-change protocol); NEW campaign-depth machinery validation (Schechter/gammaincc finite, bounded, monotone over z in [0.5, 1.5] on synthetic + committed frozen m_th map; frozen-map f_bar(1.0) ~ 0); pixelated dark-draw W_k test pinned to an explicit z_max = 0.5 window (sampler is depth-agnostic; W_k contrast lives at low z — at 1.5 the near-uniform weights drown the Pearson check in multinomial noise). Refs: population reach anchors dist(1.5, h=0.73) = 10.686 Gpc, dist(1.5, h=0.60) = 13.002 Gpc (physical_relations_test.py pins). Full fast suite: 735 passed. Closes #20. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… kernel (issue #16) User decision on issue #16 (2026-07-03): 'for the next campaign we marginalize it and test the isolated effect in parallel'. Inference-side only (re-evaluate tier) — no re-simulation, no catalogue rebuild. Old kernel (bayesian_statistics.single_host_likelihood): N(z; z_g, sigma_z_cat) [+/- 4 sigma_z_cat window] New kernel: N(z; z_g, sigma_z_eff), sigma_z_eff^2 = sigma_z_cat^2 + sigma_z_pv^2 sigma_z_pv = (1 + z_g) * SIGMA_V_PEC_KM_S / c, SIGMA_V_PEC_KM_S = 200 References: - (1+z) factor: Davis et al. (2011), arXiv:1012.2912, Eqs. (1)/(A1) (z_obs = z_cos + (1 + z_cos) v_pec/c); Davis & Scrimgeour (2014), arXiv:1405.0105, Eq. (9). - Quadrature with catalogue sigma_z: Mastrogiovanni et al. (2023), arXiv:2305.10488, Sec. IV (icarogw/GLADE+ practice). - sigma_v = 200 km/s: Fishbach et al. (2019), arXiv:1807.05667, Sec. 2.2; Chen et al. (2018), arXiv:1712.06531. LISA-EMRI precedent (Laghi et al. 2021, arXiv:2102.01708) uses 500 km/s with the (1+z) factor — kept as a systematics-budget row. Dimensional analysis: [km/s] / [km/s] = dimensionless redshift error; quadrature of two dimensionless sigmas. Limiting cases: sigma_v -> 0 recovers the old kernel exactly; z -> 0 gives sigma_z_pv = sigma_v/c = 6.67e-4 (the low-z LVK convention); at z = 1.5 the factor 2.5 gives 1.67e-3 ~ the catalogue's parse-time PV floor. Applied ONCE at function entry — all eight downstream consumption points (window bounds, Z_g renorm, prior pdf, per-host D_g, with-BH numerator/ denominator, MC proposal + sampling_pdf) flow through the single norm() object, so no double counting inside the likelihood. The catalogue z_error's parse-time PV-CORRECTION floor (0.0015) is a DISTINCT term (correction uncertainty vs residual dispersion) — documented in constants.py. Ball-tree candidate window / pruning intentionally keep the bare catalogue z_error (second-order candidate-list effect). Also: integration-testing twin mirrors the quadrature; pp_coverage gains an inert sigma_z_pv knob (default 0.0 — committed anchor runs stay bit-identical, proven by the unchanged exact-value pins). Kernel regression pins updated in this diff (protocol): at z_g = 0.1, sigma_z_cat = 0.0015 the kernel widens ~11%, integrals shift 0.2-0.5% (e.g. volume_deconv numerator 1622.007 -> 1629.370); pre-change values in the parent commit. Full fast suite: 735 passed. #16 stays OPEN pending the parallel isolated PV value-correction test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nance in DATA_INVENTORY - CHANGELOG [Unreleased]: campaign depth z=1.5 (incl. z_cut wiring, low-h d_L cap fix, ordering guard) and residual-PV marginalization (sigma_v = 200 km/s, (1+z) factor, references). - DATA_INVENTORY: new 'Galaxy Catalogue (reduced GLADE+)' section — the z_cmb 8-col full-depth CSV was previously untracked here; records schema, frame provenance (18e9608), rebuild recipe + append-mode gotcha, retired backups, and the frozen m_th map C1 coupling (unchanged by the depth constants since no CSV rebuild occurred). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, injection provenance Readiness-sweep fixes (adversarially verified, 2026-07-03) closing the window between catalogue deepening (#20) and new-injection arrival: Stale-pool gates (A2-STALE-POOL-GATE, campaign-blocking): - SimulationDetectionProbability gains expected_z_max/allow_shallow_pool: rejects pools with max z < 0.9 x expected depth and pools MIXING provenance (different z_cut stamps, or stamped + legacy files — the partial-rsync signature). Default None keeps synthetic/test pools ungated. Production constructors (evaluate + combine paths) pass expected_z_max=HOST_DRAW_Z_MAX; the combine path previously had NO check at all. - BayesianStatistics.evaluate: validate_coverage is now a HARD gate (raise below 95% coverage; --allow_low_pdet_coverage escape) instead of a warning buried in 1 of 41 task logs. - Injection writer stamps z_cut + code_rev into every row: h_inj alone cannot discriminate pre-/post-dt2 or shallow/deep pools (0.73 in every era, identical filenames). - Local 80-file pre-dt2 z_cut=0.5 pool archived to simulations/injections_RETIRED_predt2_zcut0p5_20260703/ (filename collision with the regenerated pool); DATA_INVENTORY retirement note. test_sky_selection data-gated tests now self-skip until the depth-1.5 pool lands. [PHYSICS] M_z out-of-grid policy (A2-EXTRAP): p_det with-BH-mass queries outside the injected M_z support were silently LINEARLY extrapolated (scipy fill_value=None semantics) while four comments claimed 'nearest'. Old: linear extrapolation, clipped to [0,1]. New: clamp M_z to the grid edge — true nearest, the documented Phase 44 boundary intent. Affects only out-of-support queries (rare once the pool covers the campaign range); regression test pins edge equality. Injection-loop consistency (A1/A3): - _TIMEOUT_S 30 -> 90 s, aligned with the simulation loop: timed-out events are DROPPED from the pool, so a timeout-rate correlation with (d_L, M_z) at depth 1.5 would bias the p_det grid; counter logged for smoke-test binning. - Symmetric M_z <= M.upper_limit truncation (the CRB path structurally excludes that corner via the Fisher bounds guard; the pool must match). - Pre-dt2 comments retired (93.5% z-rejection, 24/69500 horizon claim). Ops/instrumentation: - --prescreen_audit CLI flag: bypass the quick-SNR skip while logging PRESCREEN_AUDIT (quick, full, params) lines — enables the smoke-run false-negative measurement for PRE_SCREEN_SNR_FACTOR + issue #19. - --combine fast path now writes run_metadata_combine.json (TC-10: previously no git_commit/args recorded for the combine stage). - ParameterSpace d_L default 7 -> 13.1 Gpc (dist(1.5, h=0.60) = 13.0015; the old default silently rejected z >~ 0.9 events in bare constructions); dead luminostity_detection_threshold removed; completeness-plot default z_max tracks HOST_DRAW_Z_MAX. - Integration fixture pool deepened to z <= 1.5 + provenance-stamped (mirrors the production writer; exercises the gates end-to-end). mypy clean (144 files); full fast suite 734 passed / 15 data-gated skips. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… safe resubmit, provenance probes Readiness-sweep fixes TC-01..TC-15 (adversarially verified where campaign-blocking; measured anchors 2026-07-03, jobs 5732036/5735964/5735965): - evaluate.sbatch: 128 cpu/15 min -> 16 cpu/6 h placeholder (anchor: 56-76 min per h-value @ 3355 events / 16 cpus; the old pair guaranteed 38/38 TIMEOUTs + a dead chained combine); partition cpu,cpu_il; 41-value hybrid h-grid (0.655/0.665/0.675 added so the 0.005-dense window covers the 0.67 closure truth); per-task --seed (EVAL_SEED + task id) + --simulation_index metadata; per-h skip-if-output idempotency guard. - combine.sbatch: 45 -> 90 min; cluster-side figures OFF by default (RENDER_FIGURES=1 to opt in) — the figures phase is what killed job 5735965; posteriors-only anchor ~20 min. - ALL array templates (simulate/evaluate/combine/inject): private per-run CWD (/cwd with simulations + master_thesis_code symlinks) replaces the shared /simulations symlink dance — concurrent multi-seed pipelines could silently cross-contaminate CRB/ posterior writes (TC-03); merge.sbatch already --workdir-safe. - submit_pipeline.sh: --array derived from the evaluate.sbatch H_VALUES line (single grid source, TC-14); --h_true flag (threads H_VALUE, suffixes RUN_DIR) and REQUIRED --injection_pool flag (stages the pool into RUN_DIR/simulations/injections; loud failure on empty dir); submit-time posterior archiving (replaces the racy task-0 in-job archive, TC-06). - resubmit_failed.sh: TIMEOUT removed from the default state filter (gpu_h100_short tasks TIMEOUT by design — the old filter would delete ALL outputs of a healthy run); h_value recovered from run_metadata before cleanup with conflict-abort (closure-truth contamination fix); abort if the merged CRB already exists (append-duplication guard). - preflight.sh + cluster.env: O(1) catalogue fingerprint (row-1 z field: 0.001733 = z_cmb, 0.00099... = STALE z_helio — both are 8-col, so the column count cannot discriminate) + shallow-injection-pool depth probe. - datasets.yaml: seed43000_Mz/seed700 RETIRED (pre-dt2, z_cut=0.5); depth15_campaign placeholder pending regeneration. - submit_resimulate_phase50.sh: stale INJECTION_SOURCE default removed (must be set explicitly). - LAUNCHING_JOBS.md / README.md: private-CWD mental model, 2026-07-03 walltime anchors, new flags + resubmit signature. bash -n clean on all 10 shell files; 41-label convention verified against bayesian_statistics posterior filenames. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…alidity boundary measured
Referee blocker REF-P001/S006 (Paper A): sigma_z in {0.10, 0.15, 0.25} x
{bare, volume} kernels, 250 realizations x 250 events, paired seed
20260701, truths {0.62, 0.72, 0.84}.
Verdict: bare kernel rails 100% at every sigma_z >= 0.10 (unusable);
volume kernel near-nominal to sigma_z/z ~ 0.5-0.8 (cov68 0.62-0.70,
|bias| <= 0.011) and demonstrably degrades at sigma_z/z >~ 1
(sigma_z = 0.25: cov68 0.33-0.44, bias +0.04-0.06). The paper's
calibration claim gets a MEASURED validity boundary instead of the
asserted decisiveness the referee dinged. Details in SUMMARY.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The smoke run's quick-SNR false-negative measurement (--prescreen_audit, issue #19 / PRE_SCREEN_SNR_FACTOR) had no path from sbatch to the CLI. PRESCREEN_AUDIT=1 in the submitting shell now appends the flag (set -u-safe empty-array expansion). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mission Fixed BEFORE any campaign data exists (referee asserted-decisiveness critique): multi-seed MAP accuracy (2-SEM over 4 seeds), closure-truth recovery inside per-run 68% HPD, per-seed pp_coverage near-nominal (bounded by the 2026-07-03 sigma_z/z validity scan), and the verdict language for either outcome. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… resolved from package root The 46900 smoke pool came back WITHOUT z_cut/code_rev: _INJECTION_COLUMNS (the pd.DataFrame columns whitelist) silently dropped the new keys — textbook W-PRE-12 'check EVERY output writer'. Columns added + warning comment. Also _get_git_commit() ran git in the process CWD, which under the new private $RUN_DIR/cwd is not a repo (-> 'unknown' provenance); now anchored at the package's realpath so the cwd symlink resolves back to the checkout. Smoke anchors recorded (job 5739442, depth 1.5, A100): 50 events in 2:29-5:29 per task (~3-6.6 s/event SNR-only); timeouts 4/~230 (@90s); M_z > 1e6 population-bound skips ~10% (expected at (1+z) lift against Model1CrossCheck's M cap — symmetric with the CRB path by design). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion validated, variant built Isolated PV value-correction impact test (handoff §7b), phase 1: - GLADE+'s helio->CMB convention identified and VALIDATED on the 21,925,647 flag2==0 control rows: multiplicative (1+z_cmb) = (1+z_helio)(1+(v_sun/c) cos theta), Planck dipole (369.82 km/s toward l=264.021, b=48.253); residual median 5.28e-5, p99 1.98e-4. Additive and sign-flipped conventions ruled out 10-30x. Small z-dependent effective-amplitude deficit (~350-363 km/s) characterized; biases the flagged-row reconstruction by <= 2.1e-5, ~30x below the median PV correction. - Variant reduced_galaxy_catalogue_noPVcorr.csv (1.7 GB, NOT committed; git-excluded): identical to the live z_cmb catalogue except the 709,117 flag2==1 rows (3.13%, z <= 0.109) carry the frame-only transform of z_helio — i.e. the BORG PV value-correction REMOVED. Median removed |dz| = 6.30e-4 (p99 3.11e-3, max 5.19e-2). Exact row parity with the live CSV; all unflagged rows byte-identical. - Phase 2 (pending): production --evaluate on frozen seed600 events against live vs variant catalogue (identical events, same pool, --allow_low_pdet_coverage for the archived shallow baseline); posterior MAP/mean shift = the isolated PV value-correction impact, an UPPER bound for the campaign (seed600 events are all low-z). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…5% false negatives (fix #19 margin re-measurement) Old: skip events with quick(1-yr) SNR < SNR_THRESHOLD * 0.3 before the full 5-yr computation (sqrt(T) bound 0.447 + chirp margin, calibrated pre-dt2 at z <= 0.5). New: PRE_SCREEN_SNR_FACTOR = 0.0 — gate disabled; the 1-yr quick waveform is skipped entirely unless --prescreen_audit is set. Reference (measurement, not literature): depth-1.5 smoke audit, cluster job 5740080 (2026-07-03), 543 (quick, full) SNR pairs recorded via --prescreen_audit (archived at results/campaign_phase2_smoke_20260703/prescreen_audit_pairs.txt): - 3/543 FALSE NEGATIVES with full SNR >= 20 at quick SNR 0.25/1.48/5.34 (full 44.1/29.0/30.2) — sources plunging in years 2-5 accumulate SNR the 1-yr check generator cannot see; quick == full for 362/543 (plunge-within-1yr), so NO positive factor separates the populations. - A lossy gate is a selection-function inconsistency against the gate-free injection pool (the p_det grid counts those events as detectable). ~30% throughput saving not worth a 0.55% biased loss concentrated in the loud late-plunging corner. Limiting case: factor -> 0 recovers the injection-consistent selection exactly (every candidate gets the same full 5-yr SNR the pool got). Also closes the #19 remainder: the smoke measured ZERO d_L pre-screen hits at depth 1.5 (bound 11.220 Gpc = 1.05 x dist(1.5, h=0.73) vs max drawable d_L 10.686 Gpc) — margin 1.05 confirmed adequate and non-cutting on post-dt2 data. Full fast suite: 734 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ion state - datasets.yaml: depth15_campaign is CURRENT (500 files / 50k events, two-batch provenance incl. the benign mixed-code_rev note). - DATA_INVENTORY: Phase-2 campaign section — pool, design (4+2 seeds, 100x40, 41-grid, per-task eval seeds), smoke anchors, seed1000 submission (jobs 5743694-97), pre-registered criterion pointer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ependent of dev-box SSH
Submits the remaining Phase-2 pipelines (seeds 2000/3000/4000 @ 0.73 +
closures 5000 @ 0.67, 6000 @ 0.77) one at a time whenever the expanded
queue depth drops below 250 (submit cap measured between 294 and 544 on
2026-07-03). Idempotent via run-dir existence checks, so a login-node
purge or restart cannot double-submit; one automatic retry per seed;
logs to $WS/campaign_orchestrator{,_submissions}.log.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ce-staged fallback) Lets the orchestrator run from a copy on the workspace filesystem when $HOME git sync is unavailable (Lustre EIO on pack writes, 2026-07-03). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Never advance past a failed seed: retry same seed with 5-min backoff (up to 500 attempts ~ 41 h), $HOME read probe (cat of every file the submission path needs) before each attempt. - MAX_PENDING 250 -> 150 (cap measured in (294, 544]; stay clear at the low end). - Defensive cleanup of unsubmitted run dirs (no logs/) after a failed attempt so restart idempotency stays truthful. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… away-section - results/campaign_phase2_runs/watch_and_retrieve.sh: detached dev-box loop, 30-min cadence — cluster status snapshots + rsync-back of every campaign run dir and the orchestrator logs (per-poll SSH connections; nothing persistent to drop). - Runbook §4c: the three self-driving layers, Monday checklist, and the $HOME Lustre EIO situation (cluster repo pinned at b233375, git pulls blocked — code-equivalent to the campaign tag). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
combine_posteriors gained allow_shallow_pool (from the existing --allow_low_pdet_coverage flag): the combine path got the stale-pool depth gate in 3273fa5 but not the deliberate-shallow escape the evaluate path has — blocking legitimate re-evaluations of archived shallow baselines (first hit: the frozen-seed600 PV test combine). mypy clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ow plan + campaign-safety rules Fresh-session entry point while the cluster is 2FA-blocked (until Monday): first finalize the completed §7b PV analysis (post to #16), then a Workflow-orchestrated whole-codebase review — 12 dimensions with informed leads from today's recon, adversarial verification of majors, neutral-only fix wave (sim-/inference-semantic findings become issues, never landed against the running campaign), report artifact + CLAUDE.md known-bugs reconciliation. Includes the campaign-safety classification hard rule and the one running-campaign risk to check first (eval-node memory at the 4-5x larger in-RAM catalogue). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CLAUDE.md + hpc-gpu.md drifted from the code: - Known Bug #1 (unconditional cupy import) struck — guarded since 4894648 in all four GPU modules. - Known Bug #7 (bayesian_inference.py 10% distance error) marked moot — Pipeline A was removed in c1571a2; production uses per-source Cramer-Rao bounds. - Known Bug #8 (WMAP cosmology) rewritten as the documented G11 design choice (OMEGA_M=0.2726 matches Barausse-2012 M1; Planck mismatch is a tracked systematic). - Architecture bullets fixed to name bayesian_statistics.py / simulation_detection_probability.py / posterior_combination.py instead of the deleted Pipeline-A modules; bayesian_inference.py removed from the physics-trigger list. - Parameter-estimation bullet updated (5-point stencil is default since Phase 10). Docs-only; no code touched (campaign-safe). Part of the 2026-07-04 code review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Neutral cluster-script fixes (do NOT restart live jobs — deploy on Monday's git pull): - CLU-03: '|| true' on the grep-miss branches in resubmit_failed.sh and submit_pipeline.sh so the friendly empty-result / diagnostic branches actually run instead of set -euo pipefail aborting first. - CLU-06: orchestrator stop-doc uses the pkill bracket idiom so it can't signal its own ssh wrapper shell. - CLU-07: watcher run-dir glob widened to run_202607*_seed* so seeds whose run dirs are datestamped 07-07+ (orchestrator can delay a seed ~41 h under queue-cap pressure) are still mirrored. Structural cluster items (CLU-01 double-submit, CLU-02 blind drift monitor, CLU-05 --mem, CLU-09 combine afterok) tracked in issue #27 for Monday. Part of the 2026-07-04 code review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ty (issue #16) Isolated the GLADE+ peculiar-velocity VALUE correction on the frozen seed600 event set (live PV-corrected catalogue vs a frame-only variant with the correction removed on the 709k flag2==1 low-z rows). Production --evaluate, 17-value H0 grid, shared 3342-event intersection: 1D (redshift-distance): Δmean(live-noPV) = -0.0142 (-3.3σ), ΔMAP -0.010 2D (with BH mass): Δmean = +0.0012 (+0.2σ) — PV-insensitive, un-railed seed600 is the designed worst case (all-low-z, all 709k corrections). The σ_v marginalization (8568d9f) covers this; no campaign re-scope warranted. Closed #16. Commits the provenance (ANALYSIS.md, README.md, builder, queue, stats), not the 1.7 GB variant CSV or the per-h posteriors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Neutral robustness/provenance fixes from the 2026-07-04 code review (data_simulation and injection_campaign are on the simulation path, not --evaluate; the campaign runs pinned code on the cluster, so these take effect on the next git pull): - SIM-01: cancel the 90s SIGALRM in a try/finally on EVERY path (success, each exception-continue, the quick-gate continue) so no stale alarm fires in inter-iteration code and kills an unattended task. - SIM-02: scope warnings-as-errors with warnings.catch_warnings()/simplefilter so it is restored on every exit path and cannot leak into the host-refill / population sampling between iterations (and stops the global filter list growing unbounded). - SIM-03: per-(stage, exception-class) skip tally, logged at end of data_simulation, so a parameter-correlated CRB-stage drop rate (post-SNR-cut) is measurable. - SIM-04: register a SIGTERM flush handler in injection_campaign so a wall-time cap kill loses at most the events since the last periodic flush, not up to 1999 full-5yr-generator SNRs (mirrors data_simulation's existing handler). - SIM-07: bare 'raise' for the unmatched ValueError instead of raise ValueError(e). - REP-02/03: run_metadata cli_args now serialises the FULL parsed namespace via Arguments.to_dict() (captures normalization_mode, pdet_*, catalog_only, ...), and Arguments.seed is cached so it can't return a fresh random value on repeated access. Full fast suite green (739 passed), mypy clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Covers the previously-untested parse_to_reduced_catalog -> read_reduced_galaxy_catalog
round trip on a synthetic raw file: flag {1,3} filter, PV-error NaN->0.0015 floor +
quadrature fold into the redshift error, trailing integer redshift flag, the 8-column
positional reorder, and the PHI_S/THETA_S rename on read. The 8-headerless-column
contract is load-bearing for every downstream reader and drifted once before
(the .stale6col_mar28 / .zhelio_20260702 variants). 5 tests, all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dist()/cached_dist()/dist_vectorized() accept w_0/w_a but evaluate the ΛCDM-only hypergeometric form, so a wCDM call silently returned the ΛCDM result. Add a shared _reject_unsupported_wcdm() guard that raises NotImplementedError on w_0 != -1 or w_a != 0. Every production call uses the defaults (verified by grep), so this changes no computed value — it only turns a silent wrong answer into a loud one. A real wCDM distance (numerical 1/E quadrature) stays deferred to /physics-change. Also PHY-02: dist_derivative now forwards Omega_m/Omega_de/w_0/w_a to hubble_function (which implements the full CPL E(z)) instead of silently using module defaults; and the module/dist docstrings now say ΛCDM-only for the analytic distance. Full fast suite green (739 passed), mypy clean. GitHub #4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pipeline A was deleted in c1571a2; these are its last unreachable remnants (verified zero live callers by grep across source, tests, scripts, cluster): - datamodels/galaxy.py (synthetic GalaxyCatalog) + its only consumer test_benchmarks.py (the whole file was just that one slow benchmark). - constants TRUE_HUBBLE_CONSTANT, GALAXY_REDSHIFT_ERROR_COEFFICIENT, FRACTIONAL_LUMINOSITY_ERROR, FRACTIONAL_BLACK_HOLE_MASS_CATALOG_ERROR, LUMINOSITY_DISTANCE_THRESHOLD_GPC — all were galaxy.py-only after c1571a2. - handler.parse_to_reduced_catalog_with_reduced_errors — a no-op (built a local DataFrame and discarded it) with zero callers; drops the now-orphaned dist_to_redshift_error_proagation import + a docstring mention. - bayesian_statistics.single_host_likelihood_grid — a print-and-return-[] debug stub with zero callers. Production single_host_likelihood is untouched. (The dev-only single_host_likelihood_integration_testing cross-check twin is kept.) - scripts/quick_snr_calibration.py — orphaned Phase-12 one-off (SNR_THRESHOLD long settled), referenced by nothing. ruff + mypy clean, full fast suite green (739 passed). Closes the code side of #7. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
All changes are value-neutral for production (dead code or unreachable branches or comments-only): - HPC-06: power_spectral_density now raises ValueError on an unknown TDI channel instead of silently returning an all-zero PSD (which would make inner products inf/nan). Unreachable in production (callers iterate ESA_TDI_CHANNELS = 'AE'). - PHY-07: delete DarkEnergyScenario.de_equation — dead (zero callers) and wrong (divides by w_a instead of the CPL multiply; ZeroDivisionError at the fiducial w_a=0). - cosmological_model.py: fix a stale comment pointing at the removed detection_probability module (now simulation_detection_probability). - PHY-09: correct the M derivative_epsilon comment — the log-uniform midpoint is 10^5.5 ≈ 3e5 M_sun (not the stated ~3e3), and the real rationale is a phase-coherence bound for the oscillatory waveform, not Vallisneri's eps_mach^(1/4)|x|. The epsilon value (1.0) is UNCHANGED. Full fast suite green (739 passed), mypy clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sweep Neutral plotting/comment fixes from the code review (figures only; no likelihood or simulation semantics): - PLT-03/04: figure truth-lines and the CRB-derived redshift now use constants.H (TRUTH_H) instead of a hardcoded 0.73 literal, so every 'Injected' marker agrees if the fiducial changes (main.py fig01/fig02, paper_figures axvlines). - PLT-05: the interactive tension-explorer x-axis widened to [0.55, 0.88] so the top of the production 0.60-0.86 h-grid (h=0.86) is no longer rendered off-screen. - PLT-10: catalog_plots plot_glade_completeness docstring says Gpc to match its axis label (was Mpc); dropped the stale 'GalaxyCatalog plotting methods' reference. - Stale-comment sweep: TQ-04 test docstrings say HOST_DRAW_Z_MAX is 1.5 (was 0.5); the bias_investigation test_27 comment no longer points at the deleted datamodels/galaxy.py:64. Full fast suite green (739 passed), ruff + mypy clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verified-findings table (landed vs deferred-to-issue vs verified-correct), the 9 commits on this branch, the campaign risk assessment (eval-node OOM refuted), coverage gaps, and the §7b PV bound summary. Companion to issues #23-#27. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CHANGELOG [Unreleased]: Fixed/Removed/Added entries for the review's neutral changes. TODO: PHYS-6 (wCDM guard) marked done; PHYS-2/PHYS-7 marked moot (datamodels/galaxy.py deleted). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
datamodels/galaxy.py was removed as dead code; the Sphinx build failed on its lingering autodoc directive (ModuleNotFoundError, warnings-as-errors). Remove the automodule block, the architecture module-table row, and mark Limitation 8 resolved-by-removal; drop the galaxy.py file ref in Limitation 1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the hot-path d_L evaluation (~52% of H0-evaluation CPU, dominated by scipy.special.hyp2f1 which has no GPU equivalent) with a precomputed clamped cubic-spline of the comoving integral I(z), exploiting the exact factorisation d_L(z,h) = (c/H_0(h)) (1+z) I(z): E(z) and I(z) are h-independent, so one 1-D spline in z serves every h (h enters as the exact 1/H_0 prefactor). dist_to_redshift's per-call fsolve is likewise replaced by a monotone inverse spline z(u), u=(1+z)I(z). Category A numeric-parity change: the table agrees with hyp2f1 to 3.1e-10 (all h) -- tighter than hyp2f1's own ~6e-10 vs an adaptive-quad reference, and below the rel=1e-9 per-host-likelihood pin -- so the H0 MAP cannot move. Only the fiducial LCDM cosmology within the tabulated z-range [0,1.6] takes the fast path; every other input (non-fiducial cosmology, z beyond the table) falls back to the exact hyp2f1 / fsolve path bit-for-bit. Clamped BC with exact slope I'(z)=1/E(z) is required: natural BC (I''=0) is wrong at z=0 since I''(0)=-3/2 Omega_m != 0. hyp2f1 is retained as the one-time table builder and the scalar dist/cached_dist anchor (its rel=1e-12 regression pins stay green). ~8x on the d_L hot path and it removes hyp2f1 from the hot path (the GPU-portability blocker). Ref: Hogg (1999), arXiv:astro-ph/9905116, Eqs. (15)-(16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Golden-characterization regression harness pinning the Pipeline-B per-host
likelihood before the performance refactor (perf/eval-vectorization):
- test_kernel_parity.py: 22 curated single_host_likelihood cases (near/far
events, spec-z/photo-z/off-window hosts, bare vs volume-deconv z-prior, 3D
and 4D seeded-MC paths) against a stub p_det, pinned rel=1e-9.
- test_pipeline_parity.py: full evaluate() over a 3-point H0 grid on 5
synthetic events through the real RegularGridInterpolator + BallTree + pool,
pinned rel=1e-9.
Committed goldens (regen via REGEN_{KERNEL,PIPELINE}_GOLDEN=1) keep every pinned
value visible in git, so a deliberate physics change surfaces as a golden diff.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…inator
Replace the 10k-sample Monte-Carlo selection denominator D_g (the with-BH-mass
channel of single_host_likelihood, ~57% of H0-evaluation CPU) with an exact
semi-analytic estimator ("glz64"):
D_g = INTEGRAL_[z_lo,z_hi] p_gal(z) [ INTEGRAL p_det(d_L(z), M(1+z)) N(M) dM ] dz
The p_det interpolant is bilinear (method="linear") and constant-clamped in M_z,
so at fixed d_L(z) it is exactly piecewise-linear in M_z; the inner M-integral is
therefore the closed-form erf-sum over the live interpolator M_z knots
(_bh_mass_denominator_inner_m_integral, grid-agnostic -- tracks any mass_bins
change). The outer z-integral is 64-pt Gauss-Legendre over the SAME host window
[den_lo, den_hi] as the 3D denominator and the Z_g normalisation.
Why this is a physics change (Category B, values move toward truth):
- The MC carried 1-5% noise and, sampling the UNTRUNCATED z-Gaussian, was wrong
by up to +54% for low-z wide-photo-z hosts (it integrated unphysical z<0 mass
the window-normalised prior Z_g excludes). Windowing makes D_g a proper
window-averaged p_det in [0,1], consistent with the 3D path.
- On real seed400 injections the erf-sum matches a machine-precision windowed
reference to <= 2.55e-3 (worst, near-horizon; typically 1e-8..1e-4). The
likelihood-weighted per-event shift at h=0.73 is +1.7%; dominant events move
<= 1% (median). The 1D channel is unchanged (parity), so the 1D MAP is fixed.
- Deterministic (no RNG) -- strictly better than the seeded MC, ~4.5x faster;
end-to-end the full seed400 eval drops 255s -> 89s (2.86x, with the d_L table).
Re-pins: PIN_VD_BH_DEN (0.9131 -> 0.9427, kernel stub) + the kernel_parity and
pipeline_parity 4D/2D goldens (the reviewed value-update step). Correctness is
proven independently in test_bh_denominator_semianalytic (erf-sum vs a
kink-subdivided adaptive quad, rel 1e-8; constant-detectability limit).
Owen (1980) normal first-moment identity; Gray et al. (2020) arXiv:1908.06050 A.19.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Vectorize the per-(event,host) evaluation kernel over the host dimension: single_host_likelihood_batch computes all candidate hosts of a detection in one array pass, and p_Di dispatches one chunk per worker (capped at _MAX_BATCH_CHUNK hosts) instead of one starmap task per host. Eliminated per-host costs: scipy.stats.norm frozen-distribution construction (replaced by an operation-order-identical explicit Gaussian), event-level dist_to_redshift window root-finds, per-host fixed_quad node setup (module Gauss-Legendre constants, fixed_quad's exact affine map), and per-host 2,560-point p_det interpolator calls in the erf-sum denominator (single batched lookup). Value-preserving software change, not a physics change: the scalar kernel is byte-untouched and remains the reference implementation; the new test_kernel_batch_equivalence gate asserts BIT-EXACT (==) agreement between batch rows and scalar calls over the 22-regime parity grid x 7-host heterogeneous batches (770/770 values), plus chunking/ordering/cap and empty edge cases. Kernel + pipeline parity goldens pass unchanged (zero re-pins); full fast suite 831 green. Two tests mocking the old per-host starmap transport updated to the new chunk shape (same pinned numbers). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add h_values to BayesianStatistics.evaluate: one pass evaluates the whole h-list, paying the h-invariant setup once — injection pooling + P_det grid (itself h-invariant), completeness, D(h)/beta/global-selection precomputes (already h-list-native), Fisher staging, and the worker pool. Per-h posterior JSONs are written as each h completes, preserving per-h failure granularity; grid mode resets the per-h accumulators so each JSON carries exactly one likelihood per event (the canonical production shape). The single-h default path is byte-compatible: no accumulator-semantics change (the sequential-evaluate harness contract in the integration tests is preserved), same output ordering, same log lines. Cluster scripts unchanged; fusion is opt-in via the new --h_values CLI flag and can be adopted by a sibling sbatch script after the live Phase-2 campaign. Gated by test_fused_h_grid_matches_sequential: fused per-h JSON values are EXACTLY equal (==) to independent single-h evaluations over the parity H0 grid. Pipeline-parity golden unchanged; full fast suite 831 green; all slow integration tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real-data end-to-end measurement (seed400, 986 events, h=0.73, seed 0): 1D channel byte-identical; 2D channel has sparse float-path-reassociation drift at production chunk sizes (560 of ~3.0M per-host values <= 9.8e-15; 4 of 986 per-event likelihoods <= 1.4e-16) — 5+ orders under the rel=1e-9 parity contract. Soften "bit-for-bit" to the measured statement in the CHANGELOG and record the scope note in the differential test docstring. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JasperSeehofer
added a commit
that referenced
this pull request
Jul 10, 2026
…mented 713fbd1 D_g fix (+0.057→+0.025); #29 footprint 13 events, mean +0.0003, MAP unchanged run_A @fc45d1f vs 562918e run_live: 1D combined identical to 5 decimals (per-event worst rel 2.6e-8, spline-d_L tolerance). 2D moves 0.787→0.7546 mean on identical inputs = the declared Category-B semi-analytic D_g denominator fix (713fbd1, PR #31) — 57% of the open +0.057 2D residual was the MC denominator defect; remaining venue 2D residual +0.025. run_B @f29a5e7 vs run_A: hosts-present bit-identical both channels; exactly 13x17 empty->filled flips; #30 caps no-op; combine: 13 restored, 2 excluded by zero-floor (net 11); 1D MAP unchanged, mean +0.00032. Yield metric clean. Heavy per-h artifacts (8.5 GB/arm 2D) untracked per convention; combined posteriors + run_metadata force-added (redteam-artifact precedent).
Owner
Author
|
Overnight seed600 frozen-venue A/B complete — code-drift gate result (
|
JasperSeehofer
added a commit
that referenced
this pull request
Jul 12, 2026
…nt all) D1 evidence-driven (defer framing to EXP-40/campaign); D2 one combined deployment (#22->#31->#32 + #27 CLU); D3 Paper A on hold until pipeline+results satisfy, then upgrade; D4 defer 2D +0.025 to campaign (no more local bias work); PROD implement all (full truncated-normal x volume-prior + soft photo-z membership + distance-error coupling kernel, new normalization_mode, volume_deconv stays golden). Execution sequence (physics hard gate) recorded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDfQjz3Bv6oKNUQycTSDis
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack position
Part 2 of 3 in the current consolidation stack (decision D2, merge order pending):
physics/campaign-depth-pv(contains the review-branch work via PR Weekend deep code review — neutral fixes, dead-code removal, §7b PV bound #28's merge) → mainperf/eval-vectorization→ main (stacked on the review branch; merge after [PHYSICS] Phase-2 campaign: population depth z=1.5 (#20) + residual-PV marginalization (#16) #22)physics/zero-host-completion-fallback→ this branch (stacked draft PR)What
Eval-pipeline vectorization, gate-proven value-preserving (255→67 s per 17-pt grid on the profile venue, 3.8×):
afc59e9[PHYSICS] spline-table luminosity distance + inverse (value-preserving; parity-gated)e49f869kernel + pipeline parity gates for the eval refactor713fbd1[PHYSICS] exact semi-analytic with-BH-mass denominator60752ddhost-batched likelihood kernel — bit-identical, chunked dispatch51caf4dopt-in fused h-grid evaluation (--h_values) — exactly equal per h, h-invariant setup paid oncefc45d1fdocs: precise value-preservation claimVerification
562918efartifacts, identical inputs) is running overnight —results/seed600_ab_20260710/.🤖 Generated with Claude Code
https://claude.ai/code/session_01QDfQjz3Bv6oKNUQycTSDis