Feature/pandas coords#719
Closed
RobbieKiwi wants to merge 5 commits into
Closed
Conversation
added 3 commits
May 21, 2026 19:13
Collaborator
|
@RobbieKiwi many ci fails. We might only fix this in #717 |
added 2 commits
May 21, 2026 20:28
Collaborator
|
@RobbieKiwi Thanks for the PR. I would like to rewrite some of it to be more efficient: def _sanitize_pandas(arr: pd.Series | pd.DataFrame) -> DataArray | None:
names = list(arr.index.names)
if isinstance(arr, pd.DataFrame):
names += list(arr.columns.names)
if any(n is None for n in names):
return None
if isinstance(arr, pd.DataFrame):
arr = arr.stack(list(range(arr.columns.nlevels)), future_stack=True)
return arr.to_xarray()Also, we need to take care that the dim order stays deterministic (see #719). |
4 tasks
Collaborator
|
@RobbieKiwi Due to the close coupling with #710 i decided thats its easiest to combine both into a single PR. |
FabianHofmann
added a commit
that referenced
this pull request
Jun 2, 2026
…ints (#732) * fix(variables): broadcast and order pandas/DataArray bounds in coords `add_variables` had two related bugs when `lower`/`upper` were arrays: - pandas Series/DataFrame bounds missing a dimension in `coords` had the missing dimension silently dropped (#709), unlike DataArray bounds which were already broadcast. - DataArray bounds missing a dimension were expanded with `DataArray.expand_dims`, which prepends new dimensions and produces a `coords`-mismatched dimension order in the resulting variable (#706). The order depended on the type of the bounds, so scalar bounds worked but two array bounds missing the same dimension did not. Replace `_validate_dataarray_bounds` plus the downstream `as_dataarray(..., coords)` call with a single helper `_as_dataarray_in_coords`. It converts any input (pandas with named axes via `to_xarray`, otherwise via `as_dataarray`), validates the result against `coords`, expands missing dims, transposes to coords order, and reconstructs the coord variables in that order. `expand_dims` and `transpose` are no-ops when the array already matches, so scalar / full-dim DataArray bounds keep their fast path. Also fix `linopy.piecewise._broadcast_points`, which built the `expand_dims` map from a `set`, producing a hash-randomized dimension order across processes. Iterate expressions and dims in declaration order instead. Closes #706 and #709. Supersedes #710 and #719. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(variables): frame add_variables coords as source of truth Restate #706/#709's fix as a single principle in the docstring, release note, and `_as_dataarray_in_coords` helper docstring: when `coords` is provided to `add_variables`, it is the source of truth for dimensions, dimension order, and coordinate values, and `lower` / `upper` are broadcast and aligned to match. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: frame bounds fix as extending 0.7.0's coords-as-truth fix 0.7.0 already shipped "add_variables no longer ignores coords when lower / upper are DataArrays". Recast the new bullet as extending that fix to the remaining gaps (pandas bounds; dim order across bound types) so the continuity is visible from the release notes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: reword as "extend and finalize", emphasize hardening Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: rephrase as "0.7.0 made ... this release closes the two remaining gaps" Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: spell out dims/order/values in coords-as-truth bullet Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(variables): cover pandas MultiIndex bounds and dim reindex - Parametrize test_bound_broadcast_missing_dim with three additional cases: Series with MultiIndex(time, colour), DataFrame with MultiIndex columns(space, colour), and DataFrame with MultiIndex index(time, space). Exercises the `while DataFrame: unstack()` loop and the MultiIndex branch of `_named_pandas_to_dataarray`. - Add test_dataarray_coord_reorder for the same-values-different-order reindex branch (previously only the unequal-values raise was covered). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: move as_dataarray_in_coords to common.py Relocate `_as_dataarray_in_coords` and its helpers (`_coords_to_dict`, `_named_pandas_to_dataarray`) from `model.py` into `common.py`, alongside the existing `as_dataarray` they parallel. Rename to `as_dataarray_in_coords` (no leading underscore) since it is no longer file-local — other modules can import the strict-coords variant when migrating call sites. Pure relocation: no behavior change, no call-site changes beyond `add_variables`'s import. Refs #723. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(common): simplify _named_pandas_to_dataarray + cover edge branches Replace the unstack-while-loop / split named-check structure with a single up-front "all axes named" check and a single ``DataFrame.stack(level=list(range(nlevels)), future_stack=True)`` call that collapses all column levels into the row MultiIndex in one shot. Same observable behaviour, fewer moving parts, no defensive unreachable branches. Add tests covering the unnamed-axis fall-through path, the empty-coords short-circuit in ``as_dataarray_in_coords``, and the ``MultiIndex``-on-a-dim ``continue`` in the validation loop. Together with the restructure these bring the new helper code to full patch coverage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(common): only accept string axis names in _named_pandas_to_dataarray Pandas allows any hashable in ``pd.Index.names`` (tuples, ints, etc.), but only strings map cleanly to xarray dim names. Reject anything non-string up front so the pandas falls back to ``as_dataarray`` instead of producing a DataArray with an awkward non-string dim name that downstream validation would reject with a confusing "extra dimensions" error. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(common): align positional inputs to coords, with clear shape errors Inputs without their own meaningful labels — numpy arrays, polars Series, pandas with unnamed axes — fell through ``as_dataarray_in_coords`` via a short-circuit return. That meant: - The default ``dim_0`` / ``dim_1`` axis names from ``as_dataarray`` leaked into the result, so a pandas Series without an index name combined with another bound carrying a named coord produced a spurious 2-D variable. - Shape mismatches surfaced further downstream as confusing "coordinates do not match" errors against the auto-generated ``RangeIndex``. The fall-through now: (a) defaults ``dims`` to coords' keys so axes get labelled correctly; (b) runs the same validate / expand / transpose path as labelled inputs; (c) re-assigns coords from ``expected`` on the resulting DataArray so positional inputs align to coords by position. A shape mismatch surfaces as xarray's clear ``conflicting sizes`` from ``assign_coords``. MultiIndex coords are left alone (re-assigning a PandasMultiIndex emits a FutureWarning). Replaces the tautological ``test_pandas_bound_with_unnamed_axis_falls_through`` (which sneaked past by naming the coord ``"dim_0"`` to match the auto-generated dim) with ``test_positional_bound_aligns_to_coords`` that asserts actual positional alignment across numpy / Series / DataFrame, plus ``test_positional_bound_wrong_size_raises_clear_error`` for the shape-mismatch path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(sos): use var.indexes[d] for reformulated bounds; widen _coords_to_dict ``reformulate_sos1`` / ``reformulate_sos2`` built the coords for the indicator variable as ``[var.coords[d] for d in var.dims]``, which is a list of ``xarray.DataArray`` coord objects. The rest of linopy passes ``coords`` as a list of ``pd.Index``. The mix slipped through under the old short-circuit fall-through but broke once the helper started defaulting ``dims`` from ``_coords_to_dict(coords)`` — non-``pd.Index`` entries were silently dropped, so ``len(dims) < len(coords)`` and xarray raised ``different number of dimensions on data and dims: 2 vs 1``. Use ``var.indexes[d]`` instead — it returns the actual ``pd.Index`` (regular or MultiIndex) for the dim and preserves structure that ``pd.Index(coord.values, ...)`` would flatten. Also widen ``_coords_to_dict`` to accept any entry with a ``.name`` (xarray DataArrays included) so a future caller passing mixed types doesn't silently lose coords. The reformulator fix removes the only known producer of mixed-type coords; this is belt-and-suspenders. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(common): tighten _coords_to_dict to raise on non-pd.Index entries Replace the permissive ``getattr(c, "name", None)`` check with an explicit allow-list: ``pd.Index`` (named or not — unnamed silently skip as before) and unnamed sequences (``list`` / ``tuple`` / ``range`` / ``numpy.ndarray``). Any other type (notably ``xarray.DataArray``, but also ``pd.Series`` and friends) now raises ``TypeError`` with a hint to pass ``variable.indexes[<dim>]`` instead. This would have caught the SOS-reformulator bug at the source instead of letting it surface as a confusing xarray error about mismatched dim counts ten frames down. Drop ``DataArray`` from the matching ``coords`` type hints in ``model.py`` / ``expressions.py`` so the documented and runtime type sets agree. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(common): proper MultiIndex support in coords helpers (#729) - _coords_to_dict: explicitly handle pd.MultiIndex — register under .name if set, raise TypeError with guidance if .name is missing - _named_pandas_to_dataarray: use DataArray(df) directly for single-level DataFrames; reserve stack() for MultiIndex axes - as_dataarray_in_coords: validate MultiIndex dims with .equals() instead of silently skipping them - Move MultiIndex tests into dedicated TestAddVariablesMultiIndexCoords class with shared fixture * fix: apply coords-as-truth rule to mask in add_variables/add_constraints (#725) * fix(model): apply coords-as-truth rule to mask in add_variables/add_constraints Routes ``mask`` through ``as_dataarray_in_coords(mask, data.coords)`` instead of ``as_dataarray(...) + broadcast_mask(...)``, so pandas ``Series`` / ``DataFrame`` masks missing a dimension are broadcast to the variable / constraint shape (parallel to the bounds fix in the previous PR). The ``add_variables`` ``mask`` type hint widens to ``MaskLike`` to match ``add_constraints``. The deprecation announced via ``FutureWarning`` in ``broadcast_mask`` ("Missing values will be filled with False ... In a future version, this will raise an error") is now in effect: masks whose coordinates are a sparse subset of the data's coordinates raise ``ValueError`` instead of silently filling missing entries. Mask dims not in the data raise ``ValueError`` instead of ``AssertionError`` for consistency with the bounds path. ``broadcast_mask`` had no other callers and is removed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update doc/release_notes.rst Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> * refactor: unify as_dataarray; split broadcasting from coords validation (#726) * fix(model): apply coords-as-truth rule to mask in add_variables/add_constraints Routes ``mask`` through ``as_dataarray_in_coords(mask, data.coords)`` instead of ``as_dataarray(...) + broadcast_mask(...)``, so pandas ``Series`` / ``DataFrame`` masks missing a dimension are broadcast to the variable / constraint shape (parallel to the bounds fix in the previous PR). The ``add_variables`` ``mask`` type hint widens to ``MaskLike`` to match ``add_constraints``. The deprecation announced via ``FutureWarning`` in ``broadcast_mask`` ("Missing values will be filled with False ... In a future version, this will raise an error") is now in effect: masks whose coordinates are a sparse subset of the data's coordinates raise ``ValueError`` instead of silently filling missing entries. Mask dims not in the data raise ``ValueError`` instead of ``AssertionError`` for consistency with the bounds path. ``broadcast_mask`` had no other callers and is removed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: unify as_dataarray; split broadcasting from coords validation Closes #723. Folds the body of `as_dataarray_in_coords` into `as_dataarray` and extracts the contract checks into `assert_compatible_with_coords`, so linopy now has one broadcasting primitive and one validation companion. `as_dataarray(arr, coords)` aligns the result against `coords` for every input type: labels positional inputs (numpy / unnamed pandas / scalar) by position, reindexes same-values-different-order, expands missing dims, and transposes to coords order. Extra dims and disagreeing value sets on shared dims pass through unchanged, so xarray broadcasting in expression arithmetic keeps working. `assert_compatible_with_coords(arr, coords)` enforces the strict contract (`arr.dims ⊆ coords.dims`, plus exact coord-value equality on shared dims). `add_variables` and `add_constraints` now call it after `as_dataarray` for `lower` / `upper` / `mask`, replacing the deleted `as_dataarray_in_coords` helper. `_coords_to_dict` filters MultiIndex level coords out of `xarray.Coordinates` inputs so the new strict-by-default path treats `station` (and not its derived `letter` / `num` levels) as the dim. Test suite: 3698 passed (no regressions). Two existing tests were updated to reflect the new "coords is source of truth" semantics: `test_as_dataarray_with_ndarray_coords_dict_set_dims_not_aligned` (extra coord entries now broadcast in) and `test_dataarray_extra_dims` (now triggers the subset check rather than the value-mismatch check). Microbenchmark in dev-scripts/benchmark_as_dataarray.py shows flat timings vs the base branch on both add_variables-heavy and arithmetic- heavy workloads. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: dims= names unnamed coords; doctest the add_variables contract Closes a silent-failure gap in the strict coords-as-truth path: when the caller passed ``coords=[[1, 2, 3]], dims=["x"]`` to ``add_variables``, ``_coords_to_dict`` returned an empty mapping (unnamed sequences carry no dim name), so the strict checks short-circuited and bounds with extra dims or mismatched values flowed through unchecked, producing variables with frankenstein outer-joined coord values. ``_coords_to_dict`` now accepts an optional ``dims`` argument that names unnamed sequence entries by position. ``as_dataarray`` and ``assert_compatible_with_coords`` plumb it through; ``add_variables`` forwards ``kwargs.get("dims")`` to the assertions for ``lower`` and ``upper``. ``coords=[[1, 2, 3]], dims=["x"]`` now enforces the same contract as ``coords={"x": [1, 2, 3]}`` or ``coords=[pd.Index([1, 2, 3], name="x")]``. Docstring of ``add_variables.coords`` documents the contract (subset-of-dims, dim order, value match with auto-reindex, missing-dim broadcast) and includes four doctests pinning it: the extra-dim raise, the value-mismatch raise, the same-values-different-order auto-reindex, and the unnamed-coords-plus-dims opt-in. Test suite: 3698 passed (parity with the previous commit on this branch). ``pytest --doctest-modules linopy/model.py -k add_variables`` also green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: add align_to_coords with semantic validation error messages Introduce align_to_coords to wrap as_dataarray and assert_compatible_with_coords with user-facing labels (lower bound, upper bound, mask). Errors now name the argument and distinguish extra dimensions, coordinate mismatches, and conversion failures. Extend mask validation to use coords+dims= when provided. Co-authored-by: Cursor <cursoragent@cursor.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refactor(model): simplify mask align; preserve TypeError in align_to_coords Three cleanups on top of align_to_coords: - Drop the trailing ``.broadcast_like(data.labels)`` in ``add_variables`` and ``add_constraints`` mask paths. ``as_dataarray`` already expands missing dims to ``coords`` shape, so the broadcast was a no-op. - Stop overriding the caller's ``dims=`` in the ``add_variables`` mask path when ``coords is None``. The previous code stripped ``dims`` and forced ``dims=data.dims``; with ``data.coords`` being an xarray ``Coordinates`` with already-named dims, the user's ``dims`` is harmless to forward and the override was just hiding intent. Mask now goes through one ``align_to_coords`` call regardless of whether ``coords`` is supplied. - Split the exception handler in ``align_to_coords``: ``TypeError`` from unsupported input types is re-raised as ``TypeError`` (still labeled), while ``ValueError`` / ``CoordinateValidationError`` stay ``ValueError``. Preserves the original type signature for callers that want to ``except TypeError``. New test ``test_align_to_coords_preserves_type_errors`` pins the TypeError pass-through. Suite: 3703 passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: rename assert_compatible_with_coords to validate_alignment Per PR review: align on the project's `validate_*` naming convention and remove the implicit "AssertionError" connotation of `assert_*`. Pairs naturally with `align_to_coords`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * test(repr): set .name on MultiIndex coord #729 made `.name` required on `pd.MultiIndex` sequence-form coord entries (xarray needs a single dim name for the flattened index). test_repr.py was the only remaining call site missing the assignment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(types): widen _coords_to_dict to Hashable; sort with key=str `xarray.Coordinates.dims` is typed `Hashable`, so the dict-comprehension return and the `sorted()` calls in the validation message tripped mypy. The function's other branches already accept `c.name` / `dim_names[i]` (both Hashable), so widening the return type is the honest signature. Also: drop `.data` from the add_variables doctest — use the public `v.lower` property instead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(common): clarify coords-entry rules and tighten error labels (#733) * refactor(common): clarify coords-entry rules and tighten error labels Stacks on top of #732. Three small follow-ups from PR review: - Remove dead `broadcast_mask` (claimed removed in #732, was still present). - `as_dataarray`: normalize bare-tuple coord entries to lists so `coords=[(0, 1, 2)]` behaves identically to `coords=[[0, 1, 2]]` (xarray reads `(a, b)` as `(dim_name, values)` and would otherwise raise a confusing error). - `align_to_coords`: pre-validate coords via `_coords_to_dict` so TypeErrors from a bad `coords` argument propagate with their own message instead of being relabeled "<label> could not be aligned to coords: ...", which previously misdirected users to inspect the bound/mask. Docs: replace the prose paragraph in `_coords_to_dict`'s docstring with an explicit rules table covering every container form and sequence-entry case (named/unnamed `pd.Index`, `pd.MultiIndex`, bare sequences, with/without positional `dims=`). Tests: new `TestCoordsToDictRules` class in `test_common.py` mirrors the docstring table one-test-per-rule so the executable spec stays visibly aligned with the documented contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(common): allow dims= to name an unnamed pd.MultiIndex Mirrors the existing rule for unnamed pd.Index: an unnamed MultiIndex paired with a positional dims=[i] entry now gets its flat .name set to dims[i] on a shallow copy (caller's MultiIndex is not mutated). Per-level names are preserved. Removes the asymmetry between Index and MultiIndex in _coords_to_dict: both can now be named either inline (.name) or by position (dims=[i]). An unnamed MultiIndex with no positional dims still raises TypeError since xarray requires a single flat name. Adds one rule-table row and two tests (test_unnamed_multiindex_with_dims_uses_dims, test_unnamed_multiindex_without_dims_raises). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(common): scope tuple-normalize check to lists/tuples with tuple entries The previous `not isinstance(coords, Coordinates | Mapping)` form was broad and rebuilt `coords` as a fresh list on every call (even when no tuple entries were present). Switch to a positive `isinstance(coords, list | tuple)` guard with a short-circuit `any(isinstance(c, tuple) for c in coords)` check, so the comprehension only runs when there is actually a tuple to normalize. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(release_notes): restore lost bullets, surface coords breaking changes Two pre-existing Upcoming-Version bullets from master had been dropped on this branch, most likely as merge-conflict casualties: - ``LinearExpression.where`` doc + ``BaseExpression.variable_names`` entry - Mosek basic/IPM solution-inspect fix Restore both verbatim from master. Also add an explicit Breaking Changes bullet for the coord-as-truth behaviour changes that previously lived only under Bug Fixes: the mask FutureWarning -> ValueError flip, the AssertionError -> ValueError flip on extra mask dims, and the new TypeError on an unnamed pd.MultiIndex without a positional dims=[i] entry. The Bug Fixes entries still carry the migration detail; the Breaking Changes bullet points there so readers scanning by section don't miss the rename of warnings to errors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(release_notes): condense coords-as-truth entries - Merge the two Bug Fixes bullets (bounds + mask) into one. The separation read as "same fix, applied twice" without adding info; one bullet covers both with the same migration detail. - Shorten the Breaking Changes bullet — it duplicated the v0.6.3 ``FutureWarning`` and ``AssertionError`` parentheticals already in Bug Fixes; keep only the FutureWarning summary and the pd.MultiIndex addition. - Collapse the Internal as_dataarray bullet from 8 wrapped lines to one, and drop the "Validation errors name the argument" UX detail — accurate but not structural enough for a release note. No facts removed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(common): preserve MultiIndex levels when broadcasting a missing dim as_dataarray used expand_dims to add a coords dim absent from the input, which silently dropped MultiIndex level coords and left a degenerate flat index that failed to align downstream (PyPSA multi-investment). Broadcast MultiIndex-backed dims against a Coordinates template instead, falling back to expand_dims when the input already carries a level name as its own coord. Also narrow CoordsLike to drop the DataArray sequence entry (rejected by _coords_to_dict), and give align_to_coords an explicit dims parameter. * feat(common): project pandas inputs onto stacked-MultiIndex coords dims Map arr dims that name levels of a stacked-MultiIndex coords dim onto that dim: a level subset broadcasts, the full set aligns element-wise. Strict callers (add_variables/add_constraints) enforce full coverage; arithmetic keeps NaN-filling. Fixes PyPSA multi-investment regressions. * feat(common): warn on implicit MultiIndex-level projection in arithmetic The level-projection result is already convention-shaped (levels stay as aux coords on the MI dim). On the arithmetic path, flag the cases the v1 arithmetic convention will require to be explicit — subset-level broadcast and NaN-fill of uncovered entries — with EvolvingAPIWarning. Full-level, full-coverage alignment and the strict bounds path stay silent. * use lax dataarray in matmul * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refactor(common): split DataArray conversion into a 3-rung strictness ladder (#737) * refactor(common): split DataArray conversion into a 3-rung strictness ladder Replace the as_dataarray + _as_dataarray_lax pair (and the enforce_level_coverage flag) with three public entry points, each including the previous one: - as_dataarray: convert only (the former _as_dataarray_lax). Used by __matmul__, where dims missing from the constant must not be broadcast in (they would be contracted away as common dims). - broadcast_to_coords: convert + broadcast against coords (the former broadcasting as_dataarray). Used by expression arithmetic. - align_to_coords: convert + broadcast + enforce the coords contract. Used by add_variables / add_constraints (unchanged signature). The broadcasting mechanics live in one shared private core (_broadcast_core) that reports MultiIndex-level projections instead of applying policy. The entry points decide what a partial projection or coverage gap means: broadcast_to_coords warns (arithmetic convention), align_to_coords raises (coords contract). This removes the enforce_level_coverage flag and keeps validation concerns out of the broadcasting layer. No behavior changes; all call sites keep their semantics. New tests pin the ladder contrasts and the matmul dim-contraction rules. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: shorten release-notes bullet on conversion helpers Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(common): rename _broadcast_core to _broadcast_to_coords Private-twin convention: _broadcast_to_coords is the raw implementation of broadcast_to_coords (returns projection events instead of applying policy), shared with align_to_coords. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(expressions): as_expression converts constants with the broadcast rung The constraint lhs/rhs setters call as_expression(value, model, coords=self.coords, dims=self.coord_dims); forwarding those kwargs to the convert-only as_dataarray dropped the broadcasting these setters relied on (e.g. a MultiIndex-level-indexed rhs failed with an xarray AlignmentError instead of being projected onto the stacked dim). Use broadcast_to_coords instead. The other as_expression callers pass only dims (no coords), for which both rungs behave identically. Adds regression tests for the rhs setter: missing-dim broadcast and MultiIndex-level projection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(common): rename align_to_coords to strict_broadcast_to_coords The function never aligns anything — it broadcasts and raises on any mismatch it cannot resolve by broadcasting alone. "Align" is also the word that invites join= proposals (aligns take joins, broadcasts do not), so the name now states what it is: the same broadcast as broadcast_to_coords with a strict failure mode (zip(strict=True) semantics). Error messages keep the "could not be aligned to coords" wording so tests in the base branch (#732) stay untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(common): apply review polish to the strictness ladder - Document the one non-obvious policy in strict_broadcast_to_coords: partial-level broadcasts are silent (bounds-broadcast feature), unlike the warning on the broadcast rung. - Unify the first parameter name across the ladder (value -> arr). - Un-invert the warning-policy loop in broadcast_to_coords. - Rename the test whose name forced an awkward signature wrap to a behavior-oriented name (test_extra_dims_pass_broadcast_rung_fail_strict_rung). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(common): add numpydoc Parameters/Returns to the three public rungs Parameter entries carry descriptions only — types live in the function signatures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(common): unify the broadcast rungs into broadcast_to_coords(strict=...) Per review discussion: one public function instead of two, with strict as a keyword flag. - strict=True (default): any mismatch with coords raises, naming label in the error — the former strict_broadcast_to_coords. - strict=False: mismatches pass through for downstream xarray alignment — the former loose broadcast_to_coords, used by arithmetic. Strict is the default so that forgetting the flag adds safety rather than silently dropping validation. MI handling preserved exactly per mode (strict: silent partial / raise on gap; non-strict: EvolvingAPIWarning) — the scenario-B deprecation warnings land separately in #732. Call sites: model.py bounds/mask drop the long name (strict is default); arithmetic and as_expression pass strict=False explicitly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(common): require label when broadcast_to_coords is strict Restores the contract align_to_coords always had: strict-mode errors must name their subject ("lower bound could not be aligned..." rather than "Value could not be aligned..."). Enforced both statically (overloads: strict=True requires label: str, strict=False forbids it) and at runtime (TypeError). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(common): deprecate implicit MI-level projection everywhere (scenario B) Per the #737 review discussion and Fabian's decision: implicit level projection is deprecated and will raise under the v1 convention, so the EvolvingAPIWarning now fires in both modes of broadcast_to_coords — the MI check is the same for every use case: - input missing a whole level: warn (strict and non-strict) - coverage gap (level combinations without a value): warn (non-strict) / raise (strict — no downstream layer to defer the NaN to) Warning emission lives in one helper, _warn_implicit_projections, with a TODO(#738) to migrate to LinopySemanticsWarning once #717 lands. Also clarifies the MultiIndex terminology everywhere: an MI dim has *levels* and *level combinations* (one tuple per position). Docstrings carry the glossary, the coverage-gap error names the missing combinations explicitly, and "entry" is gone from messages. User-facing: add_variables / add_constraints with per-period-style bounds now emit the deprecation warning (PyPSA multi-investment). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(release_notes): surface the MI-projection deprecation and DataArray-coords breaking change - The implicit MultiIndex-level projection deprecation (scenario B) now has its own entry under Deprecations, where PyPSA users scanning for upcoming warnings will find it. - Breaking Changes gains the CoordsLike narrowing: DataArray entries in sequence-form coords raise TypeError (pass variable.indexes[dim] instead). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(common): reject unnamed-MultiIndex inputs in strict validation validate_alignment unwrapped only bare pd.MultiIndex coord entries, so Coordinates-backed (DataArray) MI dims read as non-MI and skipped the equality check. Use _as_multiindex on both sides to catch mismatches regardless of level names. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #709
Changes proposed in this Pull Request
Fix the bug with pandas coords not being broadcast
Checklist
doc.doc/release_notes.rstof the upcoming release is included.