Skip to content

feat: unlabeled-operand pairing by size (#736)#747

Merged
FBumann merged 11 commits into
feat/arithmetic-conventionfrom
feat/unlabeled-pairing
Jun 4, 2026
Merged

feat: unlabeled-operand pairing by size (#736)#747
FBumann merged 11 commits into
feat/arithmetic-conventionfrom
feat/unlabeled-pairing

Conversation

@FBumann
Copy link
Copy Markdown
Collaborator

@FBumann FBumann commented Jun 4, 2026

Closes #736. Stacked on #717 (feat/arithmetic-convention).

Draft: review-ready and design-final, but the diff is against #717, so it can't merge to master until #717 does. (Not gated by the #744 MultiIndex decision — this PR's size-pairing never touches MultiIndex level coords; its HELPER_DIMS usage is _term / _factor, orthogonal to how MultiIndexes are stored.)

Implements the last unimplemented rule of the v1 convention's coordinate-alignment intro: unlabeled operands pair with the linopy operand's dimensions by size.

The rule

numpy arrays, lists, and polars Series carry no dimension labels, so their axes adopt the operand's dims by size — the same rule for arithmetic operands and for add_variables / add_constraints bounds and masks (no positional carve-out for construction):

x = m.add_variables(coords=[a4, time5])     # dims (a: 4, time: 5)
(1 * x) + np.arange(5)                       # length-5 → pairs with `time`
(1 * x) @ np.arange(5)                       # matmul contracts `time`, result keeps `a`

Ambiguity raises (v1), pointing at the explicit fix:

y = m.add_variables(coords=[p4, q4])         # both dims size 4
(1 * y) + np.arange(4)
# ValueError: ... could pair with any of ['p', 'q'] — sizes alone cannot decide.
#             Wrap the array in an xarray.DataArray with explicit dims.
Case v1 legacy
unique size match pair by size pair positionally (agrees → silent)
size matches a non-leading dim pair by size pair positionally, warn (v1 differs)
ambiguous (shared size / square) raise pair positionally, warn
no size match raise pair positionally, warn

The rule is uniform: add_variables(coords=[a:4, time:5], lower=np.arange(5)) now resolves lower to time (positional pairing errored on a); a square/ambiguous bare-numpy bound raises asking for a DataArray wrap, instead of silently guessing.

Implementation (linopy/alignment.py)

  • _pair_axes_by_size + _dims_for_unlabeled_operand — the size-pairing with the legacy/v1 fork.
  • as_constant — normalizes degenerate operands on entry: a Python list → numpy array (lists have no numeric operators), a 0-d array → Python scalar (takes the scalar fast-path, never pairs). ConstantLike stays numeric-only.
  • _broadcast_to_coords gains unlabeled_pairing="semantic" for the arithmetic path; explicit-coords callers (add_variables) stay positional.
  • Two conversion fixes the seam exposed: as_dataarray's scalar branch and the positional fallback now exclude HELPER_DIMS, so a scalar never broadcasts over _term / _factor. (HELPER_DIMS was already the global registry; _group is transient and correctly excluded.)

Why dims=self.coord_dims was dropped from the operator calls

The arithmetic operators (expressions.py _add_constant, _apply_constant_op, to_constraint; variables.py to_linexpr; both __matmul__) previously passed dims=self.coord_dims into broadcast_to_coords. That argument pinned an unlabeled array's axes to those dims by position — which is exactly the behavior #736 replaces. While it's passed, size-pairing can never run (an explicit dims means "the caller named the axes").

So every arithmetic call site drops it. This is safe because coords=self.coords already carries the same dim information, and dims only ever affected unlabeled inputs:

  • DataArray / named-pandas operands ignore dims entirely (they carry their own labels) — unchanged.
  • Unlabeled operands (numpy / list / polars) now reach _dims_for_unlabeled_operand and pair by size — the point of the PR.

dims=self.coord_dims did do one extra thing worth calling out: it excluded helper dims (_term, _factor) from the positional labeling. Dropping it surfaced three latent HELPER_DIMS leaks (a scalar gaining _term, etc.), now fixed in the conversion layer (as_dataarray's scalar branch and the positional fallback both exclude HELPER_DIMS). as_constant on entry and UNLABELED_TYPES as the single dispatch source complete the picture; Variable operators inherit it all via delegation.

Tests

  • TestUnlabeledPairing (test_legacy_violations) — parametrized over numpy / list / polars, with @pytest.mark.legacy / @pytest.mark.v1 pairs covering pair-by-size, order-independence, ambiguity, no-match, the DataArray escape hatch, and matmul.
  • The two pre-existing unlabeled-rhs constraint tests forked legacy/v1.
  • benchmark_model example named its rhs axis (it used an ambiguous square-dim rhs — now demonstrates the convention).

Verification

Full suite under both semantics: 6458 passed, 548 skipped. mypy + pre-commit clean. convention.md §736 TODO resolved; legacy-removal.md updated.

🤖 Generated with Claude Code

)

Implements the coordinate-alignment intro of the v1 convention: numpy /
list / polars operands carry no labels, so their axes pair with the
linopy operand's dimensions by size. Ambiguity (two dims share a size,
or a square array) or no size match raises under v1, pointing to
wrap-in-a-DataArray; legacy keeps the positional pairing and warns when
the v1 result would differ or reject.

Core (linopy/alignment.py):
- _pair_axes_by_size + _dims_for_unlabeled_operand: the size-pairing,
  with the legacy/v1 fork.
- as_constant: normalize degenerate operands on entry — a Python list
  becomes a numpy array (lists have no numeric operators), a 0-d array
  unwraps to a scalar (so it takes the scalar fast-path, not pairing).
- _broadcast_to_coords gains unlabeled_pairing="semantic" for the
  arithmetic (strict=False) path; explicit-coords callers stay positional.
- Two conversion fixes the seam exposed: as_dataarray's scalar branch and
  the positional fallback now exclude HELPER_DIMS, so a scalar never
  broadcasts over _term/_factor.

Operators (linopy/expressions.py): the binary dunders call as_constant on
entry and drop the dims=coord_dims hint so unlabeled operands reach the
size-pairing; matmul pairs the contracted dim by size too. Variable
operators inherit this via delegation. UNLABELED_TYPES is the single
source of truth for the dispatch.

Tests: TestUnlabeledPairing (test_legacy_violations) — parametrized over
numpy/list/polars with legacy/v1 markers; the two pre-existing
unlabeled-rhs constraint tests forked legacy/v1. benchmark_model example
names its rhs axis (it used an ambiguous square-dim rhs).

Docs: convention.md §736 TODO resolved; legacy-removal.md lists the
positional-pairing fallback.

Full suite under both semantics: 6456 passed, 546 skipped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@FBumann FBumann marked this pull request as draft June 4, 2026 07:09
FBumann and others added 10 commits June 4, 2026 10:16
…thmetic

Per review: there's no principled reason add_variables / add_constraints
bounds should pair an unlabeled array positionally while arithmetic
operands pair by size. Positional was a coords-as-truth carryover from
#732, and it's strictly worse — it errors on cases size-pairing resolves
(`lower=np.arange(5)` against dims (a:4, time:5) now picks `time` instead
of conflicting on `a`) and silently guesses where size-pairing safely
raises.

So pairing is now by size *everywhere*, unconditionally:

- `_broadcast_to_coords` drops the strict/positional gate; the `strict`
  parameter goes back to meaning only "raise vs pass-through on mismatch".
- 0-d arrays skip pairing (no axes); they broadcast as scalars.
- The conversion is handed the normalized `expected` dict rather than the
  raw sequence-form `coords`, so coords filter by name — a sequence-form
  `coords` would otherwise zip dims to coords positionally and mis-assign
  once pairing chose a non-leading dim.

Tests: TestUnlabeledPairing gains add_variables bound cases (size-pair +
ambiguity raise). Full suite under both semantics: 6458 passed. The
convention intro already states the by-size rule without an add_variables
carve-out; _dims_for_unlabeled_operand's docstring updated to say "input"
(bounds/masks/operands), not "arithmetic operand".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…iew fixes

Address review of #747:

- Bug: to_constraint lost its docstring — `rhs = as_constant(rhs)` was
  inserted before the docstring, demoting it to a dead expression
  (to_constraint.__doc__ was None). Moved the call after the docstring.
- _broadcast_to_coords is now a 6-step pipeline; each phase is a named
  helper with a focused docstring (_label_input, _reindex_reordered_dims,
  _expand_missing_dims, _order_like_coords, _dims_for_positional_input),
  replacing the inline comment paragraphs.
- Deduplicate the matmul operand conversion: both LinearExpression and
  QuadraticExpression __matmul__ call the new
  alignment.matmul_operand_to_dataarray (one home for the contraction
  pairing).
- _dims_for_unlabeled_operand's legacy branch returns `positional`
  (len(shape) names), symmetric with the v1 `paired` return, instead of
  the full candidate list trimmed implicitly downstream.
- test: TestUnlabeledPairing.test_v1_pairs_by_size uses a stacked
  @pytest.mark.v1 decorator instead of post-hoc mark reassignment.

Full suite under both semantics: 6458 passed, 548 skipped. mypy +
pre-commit clean; restored docstring passes --doctest-modules.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The xr.DataArray(naxis, dims=["dim_0"]) wrapping is self-explanatory.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…_positional_input

Review follow-ups:
- matmul_operand_to_dataarray -> _matmul_operand_to_dataarray (internal
  helper, consistent with _dims_for_unlabeled_operand's underscore prefix).
- It no longer re-implements the "is it unlabeled? -> pair by size"
  decision; it calls _dims_for_positional_input, the single owner of
  "which dims an unlabeled operand's axes adopt". Since its expected dict
  is built from coord_dims (no helper dims), the fallback is exactly
  list(coord_dims), so behavior is unchanged. The two call paths now
  differ only in what they should: matmul converts raw, the broadcast
  pipeline runs reindex/expand/transpose.

Full suite under both semantics: 6458 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both branches of "infer dim order only when the user didn't name it" are
now covered:
- no dims + unlabeled  -> size-pair (existing TestUnlabeledPairing tests);
- explicit dims        -> honored positionally, like xarray, even when
  size-pairing would be ambiguous (new test_explicit_dims_bypass_size_pairing).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ards)

The size-pairing unification also runs under legacy (positional + a
deprecation warning), so legacy needs explicit coverage that results are
unchanged and the warnings fire correctly:

- Rename test_legacy_positional_with_warning -> test_legacy_no_divergence_
  is_silent: it never asserted a warning — it's the no-divergence case, now
  pinned as silent (LinopySemanticsWarning escalated to error).
- test_legacy_warns_when_v1_would_differ: tighten the match to "pairs by
  size instead" (the divergence message, not the ambiguity one).
- NEW test_legacy_ambiguous_pairs_positionally_with_warning: the square
  (p:4, q:4) case where v1 raises — legacy must pair positionally with the
  leading dim and warn, never raise. The strongest legacy/v1 divergence
  guard.
- NEW test_legacy_add_variables_bound_positional: the unification touched
  add_variables; pins that legacy still assigns an unlabeled bound
  positionally, producing the pre-#736 result.

Full suite under both semantics: 6461 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per review + follow-up questions:

- Cross-type uniformity is now proven, not just asserted: a shared
  UNLABELED_1D parametrization (numpy / list / polars) covers the raise
  and matmul paths too, not only the happy-path pairing test. Removes the
  inline lambda duplication.

- Multi-dim numpy coverage added:
  - 2-d square (4, 4) vs (p:4, q:4) — the (a,b)-vs-(b,a) ambiguity the
    convention calls out; exercises the multi-axis-same-size branch of
    _pair_axes_by_size that 1-d cases never reach.
  - 2-d (5, 3) no-size-match raises.
  - lower-rank operand against a 4-dim variable: a 2-d (4, 5) pairs its
    axes with the (b, c) subset by size and broadcasts over (a, d).
  - an ambiguous axis within a higher-rank operand still raises.

- Tighten the legacy divergence test's suppress(Exception) ->
  suppress(ValueError) so it pins the expected shape-conflict.

Full suite under both semantics: 6471 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cosmetic reorg (no behavioral change, same 24 tests):
- Move the `wide` fixture up beside `xy` / `square` so all three fixtures
  sit together instead of stranded mid-class.
- Order the v1 tests by surface with section comments: 1-d operands →
  multi-dim → matmul → construction. The 1-d ambiguity/no-match tests now
  sit next to their multi-dim counterparts rather than interleaved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
convention.md specifies the v1 convention; that legacy keeps the old
behavior and warns on divergence is the universal legacy-bridge pattern,
not specific to unlabeled pairing — no need to restate it here.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
It is a runtime isinstance tuple, the same category as CONSTANT_TYPES, so
it belongs with the other type definitions, not as a module-local global in
alignment.py. Derive it from an UnlabeledLike alias via get_args, mirroring
ConstantLike/CONSTANT_TYPES. Left unannotated so isinstance narrowing is
preserved at alignment.py's np.shape/np.ndim call sites.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@FBumann FBumann marked this pull request as ready for review June 4, 2026 13:10
@FBumann FBumann merged commit 11cbbca into feat/arithmetic-convention Jun 4, 2026
2 checks passed
@FBumann FBumann deleted the feat/unlabeled-pairing branch June 4, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant