diff --git a/.claude/rules/hpc-gpu.md b/.claude/rules/hpc-gpu.md index 2a49baee..61c40773 100644 --- a/.claude/rules/hpc-gpu.md +++ b/.claude/rules/hpc-gpu.md @@ -47,7 +47,7 @@ def _get_xp(use_gpu: bool) -> types.ModuleType: ### Rules -- **GPU imports must always be guarded.** Never place `import cupy as cp` at module top level unconditionally. Known issues in `decorators.py`, `memory_management.py`, `LISA_configuration.py`, `parameter_estimation.py` — fix when touched. +- **GPU imports must always be guarded.** Never place `import cupy as cp` at module top level unconditionally. All source modules are compliant as of commit `4894648` (`decorators.py`, `memory_management.py`, `LISA_configuration.py`, `parameter_estimation.py` all use the guarded `try/except ImportError` + `_CUPY_AVAILABLE` pattern) — keep new modules compliant. - **Vectorize array operations.** Never iterate over array elements in a hot path. Use vectorized `xp.*` operations (e.g., `xp.trapz(integrant / psd, x=fs)` instead of a Python loop). - **Avoid GPU-to-CPU transfers in hot paths.** Do not call `cp.asnumpy()` or `.get()` inside functions called thousands of times. Keep data on GPU until a single scalar result. - **GPU memory management.** Free GPU memory after each full simulation step (`cp.get_default_memory_pool().free_all_blocks()`). Do not call inside inner loops — the CuPy allocator reuses blocks. diff --git a/.planning/CAMPAIGN-PREP-PHASE2.md b/.planning/CAMPAIGN-PREP-PHASE2.md index 0af3becc..03f19c3a 100644 --- a/.planning/CAMPAIGN-PREP-PHASE2.md +++ b/.planning/CAMPAIGN-PREP-PHASE2.md @@ -86,6 +86,54 @@ railed baseline. After combine `5698618` completes and results are retrieved+ver - ☐ **Smoke test first**: `--tasks 2 --steps 10` (cluster skill golden rule) + the §1 pre-screen measurements, THEN full submission. +## 4b. PRE-REGISTERED "bias resolved" criterion (fixed 2026-07-03, BEFORE submission) + +Defined before any campaign data exists, per the referee's asserted-decisiveness +critique (report either outcome): + +1. **Multi-seed accuracy (primary)**: over the 4 seeds at h_true = 0.73, the + per-seed volume_deconv 1D MAP sample must satisfy + |mean(MAP) − 0.73| < 2·SEM (SEM = std(MAP)/√4). Same check for posterior means. +2. **Closure**: each closure run (0.67, 0.77) recovers its truth inside its own + 68% HPD interval (both channels). +3. **Calibration**: per-seed `pp_coverage` (G4b, volume kernel) at the campaign's + photometric width stays near-nominal — cov68 within ±0.10 of 0.68 at n=250 + realizations (binomial σ≈0.03; the 2026-07-03 σ_z/z scan bounds validity at + σ_z/z ≲ 0.8). +4. **Verdict language**: all three pass → "bias resolved at the campaign's + statistical precision"; any fail → report the measured residual with its + multi-seed uncertainty — no stronger claim. + +## 4c. Unattended weekend operation (2026-07-03 → Monday 07-07) + +Three self-driving layers (no live SSH dependency — every connection is per-poll): + +1. **Cluster login node**: `$WS/campaign_orchestrator.sh` (pid in + `$WS/campaign_orchestrator.log`) submits seeds 2000/3000/4000 @ 0.73 + + 5000 @ 0.67 + 6000 @ 0.77 (`--tasks 100 --steps 40`, pool + `$WS/injection_pool_depth15_50k`) whenever queue depth < 150; retries the + same seed through the ongoing $HOME Lustre EIO flakiness (read-probe + + 5-min backoff); idempotent on restart. +2. **Dev box**: `results/campaign_phase2_runs/watch_and_retrieve.sh` + (detached) rsyncs every run dir + orchestrator logs back every 30 min → + `results/campaign_phase2_runs/{status.log,rsync.log,run_*/}`. The §7b PV + test queue (`results/pv_correction_test_20260703/run_queue.log`) finishes + Thursday evening. +3. **Claude session watchers** (if the session survives): smoke-chain combine, + pipeline-1 sim array. + +**Monday checklist** (if anything stalled): +- `tail $WS/campaign_orchestrator.log` + `pgrep -f "campaign_orchestrator[.]sh"` + — restart with: `cd $WS && PROJECT_ROOT=$HOME/MasterThesisCode setsid --fork + bash campaign_orchestrator.sh` (idempotent). +- Cluster repo is PINNED at `b233375` (= tag code-wise): $HOME EIO blocks git + pulls (`unpack-objects failed`); commits after it are docs/ops-only. Re-sync + when the filesystem heals; report persistent EIO to the bwHPC hotline. +- Straggler arrays: `bash cluster/resubmit_failed.sh + ` (TIMEOUT excluded by default — expected on + gpu_h100_short). +- `results/campaign_phase2_runs/status.log` is the weekend flight recorder. + ## 5. Paper hooks - Paper A (`paper_a/`, drafting in flight): consumes the seed600 confirmation (§0) into diff --git a/.planning/HANDOFF-CODE-REVIEW-20260704.md b/.planning/HANDOFF-CODE-REVIEW-20260704.md new file mode 100644 index 00000000..f973872b --- /dev/null +++ b/.planning/HANDOFF-CODE-REVIEW-20260704.md @@ -0,0 +1,181 @@ +# Handoff — Weekend deep code review (2026-07-04 → 07-06) + +Fresh-session entry point. The Phase-2 campaign runs autonomously on the cluster all +weekend (2FA re-auth impossible before Monday); this session uses the time for an +in-depth, Workflow-orchestrated review of the whole codebase. User-authorized scope: +improve the local code base and push; worst case we revert; verify with local test +runs only. The user explicitly requested "a very detailed code review with workflows" +— that is the standing multi-agent opt-in for this work. + +## 0. Standing state (do NOT rediscover; verify only if something looks off) + +- Branch `physics/campaign-depth-pv` @ `8c59cce`, PR #22 open, tag + `campaign-phase2-base` = `b6bf57d`. Campaign design + away-plan: memory + `project_campaign_phase2.md` + runbook `.planning/CAMPAIGN-PREP-PHASE2.md` §4b/§4c. +- **Cluster is UNREACHABLE until Monday** (2FA ControlMaster expired; `ssh bwunicluster` + fails with "Permission denied (publickey,keyboard-interactive)" — expected, do not + retry in loops). The campaign self-drives: login-node orchestrator submits seeds + 2000–6000; local `results/campaign_phase2_runs/watch_and_retrieve.sh` (detached, + survives session clears) resumes telemetry automatically after the user's next + interactive 2FA login. Cluster repo pinned at `b233375` (+ $HOME Lustre EIO blocks + git pulls there — repair + re-sync is a MONDAY task, runbook §4c). +- **§7b PV test**: 26 local evaluations DONE incl. the h ≤ 0.785 grid extension + (detached queue; check `results/pv_correction_test_20260703/run_queue.log` says + "ALL RUNS DONE"). FIRST TASK of the new session (§2 below): finalize the analysis + and post to issue #16. Preliminary (9-value grid): 1D Δmean(live−noPV) ≈ −0.0137 + (≈2.7 posterior widths — seed600 is all-low-z, the designed upper bound); 2D was + edge-railed at 0.765, hence the extension. +- Detached local processes that must keep running: the retrieval watcher and (until + done) the PV queue. `pkill` patterns must use the `[x]` bracket idiom — a plain + `pkill -f name` matches its own wrapper shell (bit us twice on 2026-07-03). + +## 1. Weekend ground rules + +1. **Campaign-safety classification for every proposed fix** (hard rule): + - **sim-semantic** (would change what the GPU campaign is computing now, e.g. + population draws, waveforms, SNR, Fisher): PROPOSE ONLY — file an issue, never + land; re-simulation is a user decision. + - **inference-semantic** (changes posterior numbers: kernels, integrals, p_det + consumption, normalizations): allowed ONLY through `/physics-change` with pin + updates in the same diff, AND flag "re-evaluate tier vs `campaign-phase2-base`" + in DATA_INVENTORY. Default: DEFER to post-harvest unless the bug is severe. + - **neutral** (refactors, tests, guards, docs, dead-code removal, typing, plotting): + free to land with atomic commits. +2. Full fast suite green (`uv run pytest -m "not gpu and not slow"`) + `mypy` clean + before every commit; pre-commit reformat aborts the first commit attempt — re-add + and re-commit (recurring today). +3. Work on a NEW branch `review/codebase-20260704` stacked on + `physics/campaign-depth-pv` (keeps PR #22 reviewable); separate PR at the end. +4. Never touch `master_thesis_code/galaxy_catalogue/reduced_galaxy_catalogue.csv`, + `m_th_map_nside32.npy` (C1 frozen), or anything under `simulations/`. +5. GSD/GPD routing per CLAUDE.md: physics-semantic work → `/physics-change` gate; + the review itself is software work. + +## 2. FIRST: finalize §7b (30 min, before the review) + +1. Confirm `run_queue.log` ends with ALL RUNS DONE (both arms have 13 posteriors). +2. Re-run combine per arm (from each arm's `cwd/`, working dir `../simulations`, + `--combine --allow_low_pdet_coverage`), recompute MAP/mean/std per channel on the + 13-value grid; verify the 2D posteriors are no longer edge-railed (if still railed + at 0.785, extend again by 4 values — the queue script is idempotent). +3. Refine to the event INTERSECTION: the arms differ by one event (3342 vs 3343 used). + Identify it from the per-h JSONs and recompute both combined posteriors on the + shared set (production combine machinery; per-event exclusion is supported by the + JSON structure — inspect, don't guess). +4. Write `results/pv_correction_test_20260703/ANALYSIS.md` (numbers, sign convention + impact = live − noPV, upper-bound framing: seed600 all-low-z where all 709k + corrections live; campaign events at depth 1.5 far less sensitive) + commit + + post the result to issue #16 (stays open only if a re-scope is warranted; + otherwise close it citing the marginalization commit `8568d9f` + this bound). + +## 3. THE CODE REVIEW (Workflow-orchestrated, the session's main work) + +Target: every `.py` under `master_thesis_code/` (~40 files), the test suite, and +`cluster/*.sh|*.sbatch`. Roughly: fan out dimension reviewers → dedup → adversarial +verify → fix-wave (neutral fixes only, per §1) → report + issues. + +### Phase A — dimension fan-out (schema'd findings) + +One agent per dimension; each returns findings with: id, file:line, severity +(critical/major/minor/nit), campaign_safety (sim-semantic/inference-semantic/neutral), +evidence (what the code actually does — quote it), proposed_fix, effort. Point each +agent at the SPECIFIC suspicions below — they are informed leads from today's recon, +not exhaustive lists: + +1. **likelihood-core** (`bayesian_inference/bayesian_statistics.py`, ~2300 lines): + single_host_likelihood vs Gray et al. (2020) A.9/A.10/A.19; fixed_quad n=50/100 + adequacy now that windows widened (σ_v) and z_max(h) deepens with the new pool — + check node coverage over the completeness feature (f-structure lives z<0.2 while + D(h) integrates to z_max(h)≈3+ post-dt²; sweep flagged quadrature dilution); + narrow-σ_z D_g≈P_det(z_g) approximation (docstring ~line 560) degrades as σ_eff + grows — quantify; MC importance sampling (proposal=prior ⇒ weights=p_det) after + the G2d Eddington-M shift — is proposal still = prior exactly?; partition-norm + ratio-of-sums; worker-global lifecycle (init_worker vs test-set globals). +2. **selection-function** (`bayesian_inference/simulation_detection_probability.py`): + survival estimator correctness (searchsorted conventions, suffix weights, sky + bands, bandwidth_scale), h-invariance claim end-to-end, the new gates (edge cases: + empty pool, all-NaN columns), M_z clamp interaction with the 0.9/1.1 axis padding. +3. **catalogue-handler** (`galaxy_catalogue/handler.py`): parse chunking (append-mode + hazard), prune correctness, BallTree query geometry (ecliptic rotation COORD-03), + the ±1σ candidate z-window vs the σ_v-widened kernel (documented exclusion — + quantify the candidate-list loss at spec-z hosts), HostGalaxy field provenance, + **memory at depth 1.5**: the in-RAM pruned catalogue grew ~4–5× (z≤1.5 keeps far + more of the 22.6M rows) — measure RSS on load + BallTree build, times 14 workers + fork-COW on the 16-cpu eval nodes; flag if the 6h/16cpu eval jobs risk OOM + (SLURM default mem per cpu!) — this is the one finding that could matter for the + RUNNING campaign; if real, the Monday fix is an sbatch `--mem` line. +4. **completeness** (`galaxy_catalogue/pixel_completeness.py` + its consumers): + Schechter/gammaincc usage, C1 coupling sites, W_k weights, the L_cat-vs-completion + double-count tension for real z>0.5 galaxies with f≈0 (adjudicate as a Paper-B + design note + issue, NOT a code fix). +5. **simulation-loop** (`main.py`): the exception-`continue` catalogue in both loops — + which swallowed classes could bias selection (e.g. RuntimeError on loud/edge + waveforms) and should at least be counted per class; signal.alarm nesting/reset on + ALL paths; injection flush interval vs wall-cap data-loss window; provenance + completeness (which stages still lack git_commit). +6. **physics-relations & cosmology** (`physical_relations.py`, `cosmological_model.py`, + `datamodels/parameter_space.py`): adjudicate known bugs #6 (wCDM silently ignored — + propose fix or explicit NotImplementedError), #8 (WMAP fiducial — G11 says + deliberate, reconcile the CLAUDE.md entry), #9 (galaxy.py (1+z)^3 — orphaned, + fold into dead-code decision); dist_to_redshift fsolve bracketing at 13 Gpc; + derivative_epsilon choices vs five-point stencil. +7. **hpc-gpu compliance** (`parameter_estimation/`, `LISA_configuration.py`, + `memory_management.py`, `decorators.py`): **fix known code-health bug #1 this + weekend** (unconditional `import cupy` in LISA_configuration.py — guarded-import + pattern per `.claude/rules/hpc-gpu.md`; it's on the "fix when touched" list and + it's neutral); xp-pattern violations; hot-path transfers; USE_GPU threading. +8. **tests-quality** (`master_thesis_code_test/`): inventory numeric pins vs + structural tests; uncovered critical paths (handler parse_to_reduced_catalog is + UNTESTED end-to-end; ball-tree candidate selection; posterior_combination + strategies; arguments.py flag plumbing); fixture realism vs depth 1.5; marker + hygiene; the 9 data-gated skips (should assert skip-reason messages). +9. **dead-code & drift**: Pipeline A remnants (datamodels/galaxy.py GalaxyCatalog, + `bayesian_inference/bayesian_inference.py` shim + known bug #7, + LUMINOSITY_DISTANCE_THRESHOLD_GPC, `single_host_likelihood_grid` stub, + `parse_to_reduced_catalog_with_reduced_errors`, `scripts/quick_snr_calibration.py` + now moot post-gate-disable, detection.py dead scatter clips A4) — propose ONE + coherent deletion PR section with test retargeting; stale comments/line-refs; + CLAUDE.md Known Bugs section reconciliation (several entries outdated today). +10. **repro & provenance**: rng threading completeness (every stochastic site reaches + --seed?), run_metadata coverage per stage, determinism of evaluate under + num_workers variation (worker scheduling vs seeded MC — verify the G4 claim). +11. **plotting & callbacks** (`plotting/`, `callbacks.py`): `apply_style()` compliance + (user feedback memory), depth-1.5 axis ranges/defaults, figure-data provenance + (W-TOOL-11: green tests ≠ correct figure), HORIZON template consistency. +12. **cluster-ops bash** (`cluster/`): shellcheck-grade pass over ALL scripts incl. + today's edits (private-CWD blocks, orchestrator, resubmit_failed rewrite) — today + they got `bash -n` only; check quoting, set -u interactions, empty-glob behavior, + ssh BatchMode assumptions. + +### Phase B — dedup + adversarial verification + +Barrier-collect all findings; dedup by (file, line-range, topic). Then verify: +critical/major findings get an adversarial refuter (2 independent refuters for +anything inference- or sim-semantic; kill on majority-refute). Minor/nit pass +through unverified but marked. + +### Phase C — fix wave (neutral only) + report + +- Sequential fixes grouped by file-area to avoid conflicts (or worktree isolation if + parallelizing); each fix: atomic commit, tests for the fixed behavior where + meaningful, full fast suite + mypy before each commit. +- Semantic findings: GitHub issues (labels `physics`/`bug`/`design-choice` + + `paper-blocker` where apt), referenced in the report. +- Artifact: `docs/reviews/CODE-REVIEW-20260704.md` — verified findings table + (incl. refuted-with-reason), fixes landed (commit refs), issues opened, and a + short "campaign risk assessment" section (anything that could affect the running + campaign — the eval-node OOM check from dimension 3 goes here first). +- Update CLAUDE.md Known Bugs to match reality; `/pre-commit-docs` before the final + commit; push branch + open PR (base: `physics/campaign-depth-pv`). + +### Sizing guidance + +Ultracode-scale: 12 reviewers + ~2×verifiers on majors + fixers. Reviewers need +`effort: 'high'` on dimensions 1–3 (dense physics code); mechanical dimensions +(9, 11, 12) can run lower. Expect 2–3 workflow rounds: review+verify, then fix, +then a completeness-critic pass ("which files did NO reviewer actually read?"). + +## 4. Monday (unchanged, runbook §4c) + +Interactive 2FA login → watcher auto-resumes → harvest campaign per §4b criterion → +repair cluster git sync ($HOME EIO) → merge PR #22 (+ review PR) → re-sync cluster. diff --git a/CHANGELOG.md b/CHANGELOG.md index e954c47f..da16b186 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,26 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] ### Changed +- **[PHYSICS] Campaign population deepened to z = 1.5 (issue #20, decision 2026-07-03):** + `HOST_DRAW_Z_MAX` 0.5 → 1.5 (pre-dt² "horizon z ≈ 0.18, truncation exact" justification + retired); `injection_campaign` `z_cut` now derives from `HOST_DRAW_Z_MAX` (was hardcoded + 0.5 — would have left the P_det grid blind above z = 0.5); parameter-space d_L cap computed + at the lowest campaign h (0.60 → 13.0 Gpc) instead of fiducial h = 0.73 (10.686 Gpc), which + silently dropped z ≳ 1.35 events in h_true = 0.67 closure sims; explicit + `HOST_DRAW_Z_MAX ≤ max_redshift` ordering guard. `GALAXY_CATALOG_REDSHIFT_UPPER_LIMIT` + 0.55 → 1.55 (documented as unwired — the reduced CSV is full-depth, load-time depth is + `Model1CrossCheck.max_redshift`). Completeness machinery validated over z ∈ [0.5, 1.5] + (finite/bounded/monotone; frozen-map f̄(1.0) ≈ 0 — pure-completion regime). Pre-dt² + injection pools remain RETIRED; regeneration at depth 1.5 is part of the Phase-2 campaign. +- **[PHYSICS] Residual host peculiar-velocity dispersion marginalized into the host-z kernel + (issue #16, decision 2026-07-03):** `single_host_likelihood` now uses + σ_z,eff² = σ_z,cat² + ((1+z_g)·σ_v/c)² with `SIGMA_V_PEC_KM_S = 200` (Davis et al. 2011 + Eqs. 1/A1 for the (1+z) factor; Mastrogiovanni et al. 2023 §IV quadrature convention; + Laghi et al. 2021's 500 km/s kept as a systematics-budget row). Inference-side only — + no re-simulation; distinct from the GLADE+ PV-correction error already folded into the + catalogue z_error at parse time. `pp_coverage` gains an inert `--sigma-z-pv` knob + (default 0.0; committed anchor runs bit-identical). Issue #16 stays open for the isolated + PV value-correction impact test. - **Inference is now deterministic (G4):** the with-BH-mass MC denominator draws from a per-host stream derived from `(base_seed, detection_index, host_z, host_M)`; `--seed` reaches the inference layer via `evaluate(..., base_seed=...)` (default 0). @@ -142,6 +162,38 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). (`/simulations/injections`, z + SNR) for a real injected-vs-detected selection function (504k injected, SNR ≥ 20 detected) instead of gating to None. +### Fixed (2026-07-04 code review — all campaign-neutral) +- **Simulation-loop robustness** (`main.py`): the 90s SIGALRM is now cancelled in a + `try/finally` on every path (no stale alarm can fire in inter-iteration code and kill + an unattended task); warnings-as-errors is scoped with `warnings.catch_warnings()` so it + can't leak across iterations or grow the filter list unboundedly; per-(stage, exception) + skip counters make CRB-stage drop rates auditable; `injection_campaign` gained a SIGTERM + flush handler (wall-cap kill now loses ≤ one flush interval, not up to 1999 SNRs). +- **Provenance** (`main.py`, `arguments.py`): `run_metadata` now serialises the full parsed + argument namespace (`Arguments.to_dict()`) so inference-critical flags (`normalization_mode`, + `pdet_*`, `catalog_only`, …) are captured; `Arguments.seed` is cached so it can't return a + fresh random value on repeated access. +- **wCDM guard** (`physical_relations.py`, GitHub #4): `dist`/`cached_dist`/`dist_vectorized` + raise `NotImplementedError` on `w_0 ≠ -1` or `w_a ≠ 0` instead of silently returning the + ΛCDM result; `dist_derivative` now forwards its cosmology args. +- **Safety**: `LISA_configuration.power_spectral_density` raises on an unknown TDI channel + instead of returning a silent all-zero PSD (unreachable in production). +- Figure truth-lines and the CRB-derived redshift use `constants.H` instead of a hardcoded + `0.73`; the interactive tension-explorer x-axis no longer clips `h = 0.86`. + +### Removed (2026-07-04 code review) +- Pipeline-A dead code (Pipeline A itself was deleted in `c1571a2`): `datamodels/galaxy.py` + (synthetic `GalaxyCatalog`) + its lone benchmark; the galaxy.py-only constants + (`TRUE_HUBBLE_CONSTANT`, `GALAXY_REDSHIFT_ERROR_COEFFICIENT`, `FRACTIONAL_LUMINOSITY_ERROR`, + `FRACTIONAL_BLACK_HOLE_MASS_CATALOG_ERROR`, `LUMINOSITY_DISTANCE_THRESHOLD_GPC`); + `handler.parse_to_reduced_catalog_with_reduced_errors` (a no-op); the + `single_host_likelihood_grid` debug stub; `DarkEnergyScenario.de_equation` (dead + wrong); + `scripts/quick_snr_calibration.py`. + +### Added (2026-07-04 code review) +- `test_handler_catalog_io.py`: end-to-end GLADE+ reduced-catalogue writer/reader contract + test (was previously untested). + ### Fixed - `__main__.py`: force a clean process exit (`logging.shutdown()` + flush + `os._exit(0)`) at the `python -m master_thesis_code` entrypoint. The diff --git a/CLAUDE.md b/CLAUDE.md index 41488ab4..29e59a01 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -121,19 +121,18 @@ The codebase has two distinct pipelines: ### 2. Bayesian Inference Pipeline `main.py:evaluate()` -> `BayesianStatistics.evaluate()`: - Loads saved Cramer-Rao bounds from CSV -- Uses `BayesianInference` (in `bayesian_inference/bayesian_inference_mwe.py`) to compute the posterior over H0 -- `GalaxyCatalog` models the galaxy distribution and mass distribution using normal/truncnorm distributions +- Uses `BayesianStatistics` (in `bayesian_inference/bayesian_statistics.py`) to compute the posterior over H0 +- `GalaxyCatalogueHandler` (`galaxy_catalogue/handler.py`) resolves candidate hosts from the GLADE+ reduced catalogue; `SimulationDetectionProbability` supplies p_det ### Key Module Responsibilities -- **`parameter_estimation/parameter_estimation.py`** — waveform generation via `few`, Fisher matrix computation (forward-difference derivatives; 5-point stencil method exists but is not yet called — see Known Bug 4), SNR and Cramer-Rao bounds. The `scalar_product_of_functions` inner product is the computational bottleneck (PSD loop). +- **`parameter_estimation/parameter_estimation.py`** — waveform generation via `few`, Fisher matrix computation (5-point stencil derivatives, default since Phase 10), SNR and Cramer-Rao bounds. The `scalar_product_of_functions` inner product is the computational bottleneck (PSD loop). - **`LISA_configuration.py`** — LISA antenna patterns (F+, Fx), PSD, SSB<->detector frame transformations - **`datamodels/parameter_space.py`** — 14-parameter EMRI space with randomization and bounds -- **`bayesian_inference/bayesian_inference.py`** — Pipeline A (dev cross-check): `BayesianInference`, erf-based detection probability, hardcoded 10% sigma(d_L), synthetic `GalaxyCatalog`. Not used by `--evaluate`. -- **`bayesian_inference/bayesian_inference_mwe.py`** — thin re-export shim; `__main__` block runs Pipeline A standalone -- **`bayesian_inference/bayesian_statistics.py`** — Pipeline B (production): `BayesianStatistics`, `single_host_likelihood`, multiprocessing workers, helper functions. Invoked by `--evaluate`. -- **`bayesian_inference/detection_probability.py`** — `DetectionProbability` class: KDE-based detection probability with `RegularGridInterpolator` look-ups. Used by Pipeline B. -- **`cosmological_model.py`** — `Model1CrossCheck` wraps the EMRI event rate model; `LamCDMScenario`, `DarkEnergyScenario` parameter spaces. Backward-compat re-exports of `BayesianStatistics` and `DetectionProbability`. +- **`bayesian_inference/bayesian_statistics.py`** — Pipeline B (production, the only H0 pipeline): `BayesianStatistics`, `single_host_likelihood`, multiprocessing workers, helper functions. Invoked by `--evaluate`. (Pipeline A — the old `bayesian_inference.py`/`bayesian_inference_mwe.py` dev cross-check — was removed in commit `c1571a2`, 2026-05-01.) +- **`bayesian_inference/simulation_detection_probability.py`** — `SimulationDetectionProbability`: survival-estimator detection probability built from the injection pool, with `RegularGridInterpolator` look-ups. Used by Pipeline B. (Replaced the removed KDE-based `detection_probability.py`.) +- **`bayesian_inference/posterior_combination.py`** — combines per-h-value per-event posterior JSONs into the joint H0 posterior (`--combine`); zero-handling strategies and the canonical Σ log L reference implementation. +- **`cosmological_model.py`** — `Model1CrossCheck` wraps the EMRI event rate model; `LamCDMScenario`, `DarkEnergyScenario` parameter spaces. Backward-compat re-export of `BayesianStatistics`. - **`galaxy_catalogue/handler.py`** — interfaces with the GLADE galaxy catalog (BallTree-based lookups) - **`validation/pp_coverage.py`** — synthetic-universe P–P/coverage calibration harness (G4b): flat-ΛCDM tables, Malmquist selection, single-host dark-siren H₀ estimator with switchable host-z kernel ('bare' vs calibrated 'volume'). Run per-seed during campaigns. - **`constants.py`** — all physical constants and simulation configuration. Key: `H=0.73`, `SNR_THRESHOLD=20` @@ -143,15 +142,15 @@ The codebase has two distinct pipelines: ### Known Bugs to Be Aware Of #### Code health -1. **`LISA_configuration.py` unconditional `import cupy`**: still at module top level — any module that imports `LisaTdiConfiguration` is un-importable on CPU-only machines without the guarded `try/except`. Fix when that file is next touched. +~~1. **`LISA_configuration.py` unconditional `import cupy`**~~ [FIXED, commit `4894648`]: the cupy import is guarded with `try/except ImportError` + `_CUPY_AVAILABLE` in `LISA_configuration.py`, `parameter_estimation.py`, `memory_management.py`, and `decorators.py`. All source modules are CPU-importable. #### Physics / mathematics (Physics Change Protocol required) ~~4. **`parameter_estimation.py:336` Fisher matrix uses O(e) forward difference** [HIGH]~~ [FIXED Phase 10]: `use_five_point_stencil=True` is now default. Ref: Vallisneri (2008) arXiv:gr-qc/0703086. ~~5. **`LISA_configuration.py` galactic confusion noise absent from PSD** [MEDIUM]~~ [FIXED Phase 9]: `_confusion_noise()` added to `LisaTdiConfiguration`. Ref: Babak et al. (2023) arXiv:2303.15929 Eq. (17). -6. **`physical_relations.py:72` wCDM params w0, wa silently ignored** [MEDIUM]: `dist()` accepts them but passes to a hardcoded-LCDM hypergeometric function. -7. **`bayesian_inference/bayesian_inference.py` hardcoded 10% distance error** [MEDIUM]: uses `FRACTIONAL_LUMINOSITY_ERROR` instead of per-source Cramer-Rao bound from CSV. -8. **`constants.py:29-30` outdated WMAP-era cosmology** [LOW]: Omega_m = 0.25, H = 0.73; Planck 2018 best-fit is Omega_m = 0.3153, H = 0.6736. -9. **`datamodels/galaxy.py:64` galaxy redshift uncertainty non-standard scaling** [LOW]: `0.013 * (1+z)^3` has no reference; standard forms scale as (1+z). +6. **`physical_relations.py` wCDM params w0, wa silently ignored** [MEDIUM] — GitHub #4: `dist()` accepts them but passes to a hardcoded-ΛCDM hypergeometric function. The review PR (2026-07-04) adds a `NotImplementedError` guard so non-default `w_0`/`w_a` raise instead of silently returning ΛCDM. +~~7. **`bayesian_inference/bayesian_inference.py` hardcoded 10% distance error**~~ [MOOT — Pipeline A removed in `c1571a2`]. Production Pipeline B uses per-source Cramér-Rao bounds from the CSV. GitHub #5 closed. +~~8. **`constants.py` WMAP-era cosmology**~~ [RESOLVED as design choice — G11]: fiducial `OMEGA_M=0.2726`, H0=70.4 km/s/Mpc deliberately match the Barausse (2012) M1 EMRI-population cosmology (arXiv:1201.5888) for a self-consistent mock universe; the Planck-2018 mismatch is a tracked systematic in `.planning/gate/G7_systematics_budget.md`, not a bug. GitHub #6 closed. +9. **`datamodels/galaxy.py:66` galaxy redshift uncertainty non-standard scaling** [LOW] — GitHub #7: `0.013 * (1+z)^3` has no reference; **this file is dead** (imported only by `test_benchmarks.py`; production uses `galaxy_catalogue/handler.py`). Slated for deletion in the review PR. --- @@ -183,7 +182,6 @@ Any edit to these files that modifies a computed value (not just refactoring/typ - `LISA_configuration.py` - `parameter_estimation/parameter_estimation.py` - `datamodels/galaxy.py` -- `bayesian_inference/bayesian_inference.py` - `bayesian_inference/bayesian_statistics.py` - `bayesian_inference/simulation_detection_probability.py` - `cosmological_model.py` diff --git a/DATA_INVENTORY.md b/DATA_INVENTORY.md index 5066fd0f..845bd75a 100644 --- a/DATA_INVENTORY.md +++ b/DATA_INVENTORY.md @@ -53,6 +53,49 @@ corresponding tier before reporting results. --- +## ⚠️ Injection pools RETIRED — 2026-07-03 (depth-1.5 campaign prep) + +All pre-dt² / z_cut = 0.5 injection pools are **RETIRED** by the issue-#20 depth change: +local `simulations/injections/` (80 files) moved to +`simulations/injections_RETIRED_predt2_zcut0p5_20260703/`; cluster pools +(`seed43000_Mz`, `seed700`) marked retired in `cluster/datasets.yaml`. The campaign +regenerates a single-h (h_ref = 0.73) pool at z_cut = 1.5 with the **same filenames** — +never mix the eras. Guards: injection rows now carry `z_cut` + `code_rev` provenance +columns; `SimulationDetectionProbability` rejects shallow/mixed pools +(`expected_z_max`, readiness sweep A2-STALE-POOL-GATE); `--evaluate` hard-fails below +95% P_det grid coverage (`--allow_low_pdet_coverage` to override deliberately). + +--- + +## Phase-2 Campaign (2026-07-03 → , tag `campaign-phase2-base` = `b6bf57d`) + +| Item | Value | +|---|---| +| **Injection pool** | `$WS/injection_pool_depth15_50k` — 500 files / 50 000 events, z_cut = 1.5, single h_ref = 0.73, provenance-stamped (see `cluster/datasets.yaml` `depth15_campaign`) | +| **Design** | 4 seeds @ h_true = 0.73 (BASE_SEED 1000/2000/3000/4000) + closure 0.67 (5000) / 0.77 (6000); `--tasks 100 --steps 40` (~4k detections/seed target); volume_deconv; 41-value hybrid h-grid; per-task eval seeds `SEED·1000 + task` | +| **Smoke** | run_20260703_seed900 (jobs 5740080-83) — sim/merge validated; prescreen audit 543 pairs → quick gate DISABLED (`b6bf57d`); anchors: ~42 s/detection (GPU), injections 3-6.6 s/event | +| **Submitted** | seed1000: jobs 5743694-97 (2026-07-03 ~12:45Z). Remaining seeds staggered against the ~300-job submit cap | +| **Criterion** | pre-registered in `.planning/CAMPAIGN-PREP-PHASE2.md` §4b BEFORE submission | + +--- + +## Galaxy Catalogue (reduced GLADE+) + +The single on-disk input the whole pipeline shares — previously untracked here. + +| Property | Value | +|----------|-------| +| **File** | `master_thesis_code/galaxy_catalogue/reduced_galaxy_catalogue.csv` (headerless, **1.68 GB**, 22 641 048 rows) | +| **Schema** | **8 columns** (order = `_reduced_catalog_column_names()`): RA_deg, Dec_deg, B_mag, **z_cmb**, z_error (PV-correction error folded in quadrature, 0.0015 floor), stellar_mass, stellar_mass_err, z_flag (1=photo-z, 3=spec-z; trailing) | +| **Frame** | **z_cmb** (CMB frame) since `18e9608` (2026-07-02 rebuild; 99.9% rows shifted, median \|Δz\| 6e-4 — `.planning/gate/GATE_SIGNOFF.md:27`) | +| **Depth** | **Full-depth** (no z cut in the writer; max z ≈ 7.03). Effective load-time depth = `Model1CrossCheck.max_redshift` = 1.5 via `_get_pruned_galaxy_catalog`. `GALAXY_CATALOG_REDSHIFT_UPPER_LIMIT` is documentation-only | +| **Rebuild** | `results/commission_20260701/scratch/rebuild_catalog.py` from repo root; **move the old CSV aside first** (writer appends, `mode="a"`); ~77 s full GLADE+ pass on the dev box | +| **Source** | `master_thesis_code/galaxy_catalogue/GLADE+.txt` (6.4 GB, dev box ONLY — cluster cannot rebuild; staging is rsync of the reduced CSV per `/cluster` skill) | +| **Superseded** | `.zhelio_20260702` (z_helio 8-col, 2026-07-01), `.stale6col_mar28` (6-col) — backups next to the live file, RETIRED | +| **Coupled artifact** | `m_th_map_nside32.npy` (frozen per-pixel m_th, C1: byte-identical on injection + inference sides). Built from the full flag-{1,3} catalogue → **unchanged by the 2026-07-03 depth constants** (no CSV rebuild occurred); MUST be regenerated atomically on both sides if the CSV content ever changes | + +--- + ## Dataset Registry ### phase45-seed200-20260501 *(current canonical, post-Tier-3 fix)* diff --git a/TODO.md b/TODO.md index c4370b9b..55a0a1a7 100644 --- a/TODO.md +++ b/TODO.md @@ -74,7 +74,7 @@ reference, dimensional analysis, limiting case). Fix: `cv_grid = 4π · (c/H₀)³ · I(z)² / E(z)`. Ref: Hogg (1999) arXiv:astro-ph/9905116 Eq. (27). Also renamed all methods from `comoving_volume` → `comoving_volume_element` for clarity. -- [ ] **PHYS-2 [P0, S]** Fix `setup_galaxy_mass_distribution` NormalDist branch in `datamodels/galaxy.py:292` +- [x] **PHYS-2 [P0, S]** ~~Fix `setup_galaxy_mass_distribution` NormalDist branch in `datamodels/galaxy.py:292`~~ MOOT — `datamodels/galaxy.py` (Pipeline-A synthetic catalog) deleted in the 2026-07-04 code review; production uses `galaxy_catalogue/handler.py`. Sigma uses hardcoded `10**5.5` instead of `galaxy.central_black_hole_mass`. `append_galaxy_to_galaxy_mass_distribution` (line 230) already uses the correct value. The truncnorm branch also doesn't pass `loc`/`scale` to scipy (inherits defaults @@ -94,13 +94,13 @@ reference, dimensional analysis, limiting case). with `delta_luminosity_distance_delta_luminosity_distance` from the `Detection` dataclass. Requires threading per-detection error information from `EMRIDetection` into `BayesianInference`. -- [ ] **PHYS-6 [P2, S]** Fix or document silent wCDM fallback in `physical_relations.py:72` - `w_0`, `w_a` params are accepted but `lambda_cdm_analytic_distance` ignores them entirely. - Either (a) remove args and document ΛCDM assumption, or - (b) fall back to numerical integration via `hubble_function()` when `w_0 ≠ -1` or `w_a ≠ 0`. - Ref: Hogg (1999) arXiv:astro-ph/9905116 Eq. (14–16). +- [x] **PHYS-6 [P2, S]** ~~Fix or document silent wCDM fallback in `physical_relations.py:72`~~ + DONE (2026-07-04, commit `8c789a6`, GitHub #4): `dist`/`cached_dist`/`dist_vectorized` + now raise `NotImplementedError` on `w_0 ≠ -1` or `w_a ≠ 0` instead of silently returning + the ΛCDM result. A real wCDM numerical implementation remains option (b), deferred to + `/physics-change`. -- [ ] **PHYS-7 [P2, S]** Document or fix galaxy redshift uncertainty in `datamodels/galaxy.py:64` +- [x] **PHYS-7 [P2, S]** ~~Document or fix galaxy redshift uncertainty in `datamodels/galaxy.py:64`~~ MOOT — `datamodels/galaxy.py` deleted in the 2026-07-04 code review (dead Pipeline-A code); production z-errors come from the GLADE+ catalogue + the σ_v PV term. Current `0.013 * (1+z)³` caps at z ≈ 0.048, meaning almost all galaxies (z up to 0.55) use the capped value of 0.015. Standard forms: photometric `σ_z = 0.05(1+z)`, spectroscopic `σ_z = 0.001(1+z)`. Add citation or switch to standard form. diff --git a/cluster/LAUNCHING_JOBS.md b/cluster/LAUNCHING_JOBS.md index cf10368b..3cbe8627 100644 --- a/cluster/LAUNCHING_JOBS.md +++ b/cluster/LAUNCHING_JOBS.md @@ -16,15 +16,17 @@ There are **two distinct directories**, and confusing them causes most failures: | **RUN_DIR** | this run's **output** (logs, CSVs, posteriors) | `$WORKSPACE/run_YYYYMMDD_seedS/` | The package uses **relative paths from the current working directory**: -- it reads the catalog from `./master_thesis_code/galaxy_catalogue/`, +- it reads the catalog from `./master_thesis_code/galaxy_catalogue/` (handler.py:24), - it reads/writes `./simulations/…`. -So every job does the same dance: **`cd $PROJECT_ROOT`** then -**`ln -sfn $RUN_DIR/simulations $PROJECT_ROOT/simulations`** — run from the code, -but redirect `./simulations` to this run's output. Output therefore lands in -`$RUN_DIR/simulations/`. (Because there is one shared symlink, avoid running two -jobs from the same PROJECT_ROOT with different RUN_DIRs *interactively* at once; -SLURM tasks each re-point it at start, and all point to the same RUN_DIR per job.) +So every batch job runs from a **private per-run CWD** (TC-03): it `cd`s into +`$RUN_DIR/cwd/`, which holds two symlinks — +`simulations → $RUN_DIR/simulations` and +`master_thesis_code → $PROJECT_ROOT/master_thesis_code`. Code and catalog come +from the one repo; output lands in `$RUN_DIR/simulations/`. Because each run +owns its CWD, **concurrent runs with different RUN_DIRs are safe** — there is +no shared `$PROJECT_ROOT/simulations` symlink to fight over anymore. +(`merge.sbatch` needs no CWD tricks — it uses absolute `--workdir` paths.) **Env threading:** submit wrappers pass everything the sbatch needs via `sbatch --export=ALL,RUN_DIR=…,BASE_SEED=…`. The sbatch validates them and falls @@ -38,11 +40,11 @@ job `source cluster/modules.sh` (which also exports `$WORKSPACE`, `$PROJECT_ROOT ## 2. Partition cheat-sheet -| Partition | Use | Limits | +| Partition | Use | Limits / anchors (2026-07-03) | |---|---|---| | `gpu_h100_short` | production GPU sim (tasks are time-capped, backfills fast) | 30-min wall, 1 GPU/task | -| `gpu_a100_short` | GPU smoke tests | 5-min wall | -| `cpu_il` | inference / merge / combine | up to 128 cpus, ~15 min/h-value | +| `gpu_a100_short` | injection campaigns (`inject.sbatch`) + GPU smoke tests | `inject.sbatch` requests a 30-min wall; the smoke test uses 5 min | +| `cpu,cpu_il` | inference / merge / combine | evaluate: **56–76 min per h-value @ 3355 events / 16 cpus** (jobs 5732036, volume_deconv; 6h pre-smoke budget, re-size after smoke); combine: **~20 min posteriors** + 90-min budget (job 5735965 anchor); figures rendered locally (`RENDER_FIGURES=0`) | | `dev_gpu_h100` / `dev_*` | quick queue for testing | short wall, fast start | Seed convention everywhere: **per-task seed = `BASE_SEED + SLURM_ARRAY_TASK_ID`** @@ -57,12 +59,19 @@ Run all of these **from `~/MasterThesisCode` after `source cluster/modules.sh`.* ### 3a. Simulation → CRB (+ auto merge → evaluate → combine) One command chains simulate (GPU) → merge (CPU) → evaluate (CPU) → combine: ```bash -bash cluster/submit_pipeline.sh --tasks 100 --steps 50 --seed 42 +bash cluster/submit_pipeline.sh --tasks 100 --steps 50 --seed 42 \ + --injection_pool "$WORKSPACE/injection__seed/simulations/injections" # creates $WORKSPACE/run_YYYYMMDD_seed42/ ; prints all job IDs + a sacct line ``` - `--tasks` = GPU array size, `--steps` = EMRI iterations/task, `--seed` = base seed. +- `--injection_pool` (required unless `--no_injections`) links the pool's + `injection_h_*.csv` into `RUN_DIR/simulations/injections/` at submit time, so + evaluate's p_det grid uses exactly the intended pool (see `cluster/datasets.yaml`). +- `--h_true V` sets the injected truth for closure runs (default 0.73); a + non-default truth is embedded in the run-dir name (`run_YYYYMMDD_seedS_h0p67`). - Dependency chain: simulate → merge (`afterany`, tolerates task timeouts) → - evaluate (`afterok`, 38-point h-grid 0.60–0.86) → combine (`afterok`). + evaluate (`afterok`, h-grid parsed from `evaluate.sbatch` — currently 41 + points 0.60–0.86) → combine (`afterok`). - **Test small first:** `--tasks 2 --steps 10`. ### 3b. Injection campaign → P_det pool @@ -79,7 +88,8 @@ bash cluster/submit_injection.sh --tasks_per_h 80 --steps 900 --seed 12345 Usually part of 3a, but to (re-)evaluate an existing run: ```bash RUN=$WORKSPACE/run_20260516_seed400_phase50 -sbatch --parsable --array=0-37 \ +# --array must match the H_VALUES count in evaluate.sbatch (currently 41 → 0-40) +sbatch --parsable --array=0-40 \ --output="$RUN/logs/evaluate_%A_%a.out" --error="$RUN/logs/evaluate_%A_%a.err" \ --export=ALL,RUN_DIR="$RUN" cluster/evaluate.sbatch # then combine: @@ -166,13 +176,25 @@ inline wrapper that makes `RUN_DIR` and passes `--export`. ## 6. Re-run safety & idempotency - **Skip-if-output** per unit (evaluate/combine already do this): guard on the - target file so resubmits don't redo finished work. -- **Archive-then-write**: `evaluate.sbatch` task 0 archives existing - `posteriors*/` to `simulations/archive/eval_/` before a fresh sweep — so a + target file so resubmits don't redo finished work. `evaluate.sbatch` exits 0 + per-task if its `h_