refactor(benchmarks): decouple CI baseline from analytics + unify measurements() (split prep)#34
Draft
FBumann wants to merge 2 commits into
Draft
refactor(benchmarks): decouple CI baseline from analytics + unify measurements() (split prep)#34FBumann wants to merge 2 commits into
FBumann wants to merge 2 commits into
Conversation
…cs layer
Prep for splitting the local benchmarking (memory, sweep, plotting, …) into a
standalone package — makes the CodSpeed CI baseline import-clean of the
analytics layer:
- Move `spec_param_id` (the `<name>-<axis>=<value>` param-id convention) from
`snapshot.py` into `registry.py`. It's used by the core (`registry.param_ids`,
`test_to_solver`) and was the only core → analytics import.
- Stop eagerly importing `bench` in `benchmarks/__init__`; `models`/`patterns`
stay (they register specs). `from benchmarks import bench` still works on
demand (submodule import).
Result: importing the package + the three CodSpeed test modules pulls zero of
{memory, sweep, plotting, snapshot, bench, cli}; the dependency arrow now only
points analytics → core. No behaviour change — collection, harness tests (incl.
the memory↔pytest id alignment), and the memory CLI are all unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pytest drivers and the memray engine each re-derived "what runs + its node id" for every phase — a duplication that test_memory_id_alignment.py existed to band-aid. Collapse it into a single source of truth: phases own a phase_cases(phase) -> Iterable[PhaseCase], where each case is a context manager (setup → measured action → teardown). - phases.py: phase_cases + the per-phase case CMs (build/matrices/to_lp/netcdf/ to_solver/pipeline) + PHASE_NODE. Setup (build, scratch files) runs untimed/ untracked before the yielded action; the build is excluded for export phases and included for build/pipeline — uniformly, by construction. - drivers: each test_<phase> is now a thin parametrize-over-phase_cases + the shared conftest.run_case. Node ids are byte-identical (verified by an empty `pytest --co` id diff), so CodSpeed baselines are untouched. - memory.run_phase consumes the same phase_cases; _measurements and _phase_tag are gone, and the memray engine now measures every available solver (was highs-only) with ids matching pytest's. - test_memory_id_alignment.py deleted — the ids are now the same object, not two strings hoped to match. Also makes `pipeline` a first-class phase (build → matrices → lp in one region, build included): selectable via `--phase pipeline` for both metrics, with a new test_pipeline.py for timing. It stays opt-in (deselected unless `--pipeline`) and out of CI (not in CODSPEED_MODULES). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5 tasks
Merging this PR will not alter performance
Performance Changes
Comparing Footnotes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TODO (human): one line on why.
Note
Generated by AI (Claude).
Foundation for splitting the local analytics (plotting / sweep / memray engine / CLI) out of linopy into a standalone package — does the two structural changes that make that split clean, with no behaviour change.
Changes
1. Decouple the CI baseline from the analytics layer.
The CodSpeed baseline (
registry+models/patterns+phases+test_*.py+conftest) now imports none of{memory, sweep, plotting, snapshot, bench, cli}. Two things were in the way:spec_param_id(the<name>-<axis>=<value>convention) moved fromsnapshot.pyintoregistry.py— it's a core concept used by the drivers.benchmarks/__init__no longer eagerly importsbench(from benchmarks import benchstill works on demand).Result: importing the package + the three CodSpeed test modules pulls in zero analytics; the dependency arrow only points analytics → core.
2. One
measurements()contract for timing + memory.The pytest drivers and the memray engine each re-derived "what runs + its node id" per phase — a duplication
test_memory_id_alignment.pyonly band-aided. Now phases ownphase_cases(phase) -> Iterable[PhaseCase](each case a setup→action→teardown context manager); the drivers parametrize over it and the memray engine loops over it.pytest --codiff) → CodSpeed baselines untouched._measurements/_phase_taggone; the memray engine now measures every available solver (was highs-only); the alignment test is deleted (ids are the same object now).pipelinebecomes a first-class opt-in phase (--phase pipeline, both metrics; out of the default run + CI).Verify