fix: derive the extends merge policy from each key's validator

architecture/supported-subset.md

@@ -234,26 +234,42 @@ slot (`architecture/glossary.md`).
     `entrypoint` (argv replaced wholesale, never concatenated) — also
     covers unknown keys, which `validate()` then rejects downstream exactly
     as it would without `extends`.
-  - **Normalization before merge:** list-form `environment` and list-form
-    `depends_on` (a bare service-name list) are normalized to mappings
-    before the mapping-merge; scalar-form `tmpfs`/`env_file` are normalized
-    to a one-element list before the concatenation.
+  - **Normalization before merge — a merge never widens the gate.** A merged
+    side is normalized only through a form that key *actually has*: list-form
+    `environment`, `labels`, `annotations`, `extra_hosts` and `depends_on`
+    become mappings before the mapping-merge, and scalar-form `tmpfs`/`env_file`
+    become a one-element list before the concatenation. Nothing else is
+    coerced. This matters because `resolve_extends` runs **ahead of**
+    `validate()`: normalizing a form a key does not have would hand the gate a
+    document it would have
+    refused standalone. So `ulimits` and `healthcheck` (no list form) refuse a
+    list, and every list-only key (`cap_add`, `cap_drop`, `security_opt`,
+    `devices`, `group_add`, `volumes`, `secrets`, `configs`) refuses a bare
+    scalar — exactly as each does outside `extends`. For a **registry** key the
+    accepted forms are read from the key's own validator (`spec.validate` runs
+    on both sides before `spec.merge`), so those cannot drift from the gate by
+    construction. The **structural** keys have no `KeySpec`, so their accepted
+    forms are declared in `extends.py` (`_STRUCTURAL_*`, `_SCALAR_FORM_KEYS`)
+    and kept honest by a test asserting, for every mergeable key, that a form
+    the gate refuses standalone is refused through `extends` too.
+    `extra_hosts`' list form is `host:ip` — colon-separated, split on the
+    *first* colon so an IPv6 address survives (`myhost:::1` →
+    `{myhost: ::1}`).
 - **Refused loudly:** cross-file `extends: {file: ..., service: ...}`; a
   bare-string (or any other non-mapping) `extends`; an unrecognized key
   under `extends` other than `service`; a non-string `service`; a `service`
-  naming one that doesn't exist; and a merge across incompatible forms (a
-  mapping-merge key that is neither a mapping nor list-form
-  `environment`/`depends_on`; a sequence-concatenate key that is neither a
-  list nor a scalar string).
-- **Divergences from Compose:** `environment`/`depends_on` accept list form
-  directly in `extends.py`'s own merge (`_as_mapping`); `extra_hosts`/
-  `healthcheck` do not — list form on a merged side is refused for those two.
-  `labels`/`annotations`/`ulimits` instead route through their `SERVICE_KEYS`
-  merge policy (`_merge_map`), which *does* coerce list form, via the same
-  `pairs_to_mapping` normalizer `environment` uses — both paths reject a
-  non-string list element identically, since it's the one function they
-  share. Short-form `volumes` are concatenated rather than merged by target
-  path; podman resolves duplicate mounts at run time. Referenced resources
+  naming one that doesn't exist; and a merge across incompatible forms — a
+  side whose form that key does not have. A registry key raises with its own
+  validator's message (`'cap_add' must be a list`), the same message the value
+  produces standalone; a structural key raises `cannot merge '<key>' across
+  incompatible forms`. A malformed *form* is reported against the service the
+  value belongs to — the base, when it is the base's value at fault, not the
+  service extending it. (One exception: an explicit `null` on a mergeable key
+  is valid standalone but refused on merge, and a null in the base is reported
+  against the extending service.)
+- **Divergences from Compose:** short-form `volumes` are concatenated rather
+  than merged by target path; podman resolves duplicate mounts at run time.
+  Referenced resources
   (top-level `volumes`, `networks`, `secrets`, `configs`) are not
   auto-imported — as in Compose, the extending service must declare what it
   needs.

compose2pod/extends.py

@@ -3,16 +3,23 @@
 from typing import Any
 
 from compose2pod.exceptions import UnsupportedComposeError
-from compose2pod.keys import SERVICE_KEYS, concat_list, pairs_to_mapping
+from compose2pod.keys import SERVICE_KEYS, as_list, pairs_to_mapping
 
 
 # Merge policy for keys with a SERVICE_KEYS KeySpec comes from spec.merge (see
 # _merge below); these two sets cover only the remaining structural keys, which
-# have no KeySpec. Unifying structural-key merge policy is deferred — see
-# decisions/2026-07-12-reject-structural-key-registry.md's revisit trigger.
+# have no KeySpec. Both categories obey one rule: a merge may normalize a value
+# only through a form that key actually has, so it can never accept a shape the
+# gate would reject standalone.
 _STRUCTURAL_MERGE_KEYS = {"environment", "extra_hosts", "healthcheck", "depends_on"}
 _STRUCTURAL_CONCAT_KEYS = {"secrets", "configs", "volumes", "tmpfs", "env_file"}
 
+# The only concat keys Compose gives a bare-string form. The gate accepts a
+# scalar for exactly these two and requires a list for the rest, so only these
+# two may be normalized scalar -> [scalar] on a merged side; normalizing any
+# other would accept a shape the gate refuses standalone.
+_SCALAR_FORM_KEYS = {"tmpfs", "env_file"}
+
 
 def _extends_target(name: str, ext: Any) -> str:  # noqa: ANN401 - Compose values are untyped
     """Return the referenced service name, after refusing cross-file and malformed forms."""
@@ -39,19 +46,54 @@ def _extends_target(name: str, ext: Any) -> str:  # noqa: ANN401 - Compose value
     return service
 
 
+def _extra_hosts_to_mapping(name: str, value: list[Any]) -> dict[str, Any]:
+    """Normalize list-form `extra_hosts` ('host:ip') to a mapping.
+
+    Separated by a colon, not '=', so `pairs_to_mapping` would mangle the whole
+    entry into a single `{'host:ip': None}` key. Split on the *first* colon only:
+    an IPv6 address is itself full of them (`myhost:::1` -> `{'myhost': '::1'}`).
+    """
+    result: dict[str, Any] = {}
+    for item in value:
+        if not isinstance(item, str) or ":" not in item:
+            msg = f"service {name!r}: extra_hosts entries must be 'host:ip' strings"
+            raise UnsupportedComposeError(msg)
+        host, _sep, address = item.partition(":")
+        result[host] = address
+    return result
+
+
+def _as_concat_list(name: str, key: str, value: Any) -> list[Any]:  # noqa: ANN401 - Compose values are untyped
+    """Normalize a structural concat key's value to a list, without widening the gate.
+
+    `keys.as_list` turns a bare string into a one-element list, which is right
+    for `tmpfs`/`env_file` (Compose gives them a scalar form and the gate accepts
+    one) and wrong for `volumes`/`secrets`/`configs`, where the gate requires a
+    list -- normalizing there would let `extends` accept a scalar the same
+    document would be refused for standalone.
+    """
+    if isinstance(value, str) and key not in _SCALAR_FORM_KEYS:
+        msg = f"service {name!r}: '{key}' must be a list"
+        raise UnsupportedComposeError(msg)
+    return as_list(name, key, value)
+
+
 def _as_mapping(name: str, key: str, value: Any) -> dict[str, Any]:  # noqa: ANN401 - Compose values are untyped
     """Normalize a structural mapping-merge key's value to a mapping.
 
-    Deliberately stricter than `keys.pairs_to_mapping`: list form is accepted
-    only for `environment` and `depends_on`, the two keys Compose actually
-    defines a list form for. `extra_hosts`/`healthcheck` in list form on a
-    merged side are refused as an incompatible form rather than coerced.
+    List form is accepted for exactly the keys Compose defines one for --
+    `environment`, `extra_hosts`, `depends_on` -- each through the normalizer
+    that key's own list form actually needs. `healthcheck` has no list form, so
+    a list is refused rather than coerced: a merge must not accept a shape the
+    gate would reject standalone (see `_merge`).
     """
     if isinstance(value, dict):
         return value
     if isinstance(value, list):
         if key == "environment":
             return pairs_to_mapping(name, key, value)
+        if key == "extra_hosts":
+            return _extra_hosts_to_mapping(name, value)
         if key == "depends_on":
             for dep in value:
                 if not isinstance(dep, str):
@@ -62,17 +104,32 @@ def _as_mapping(name: str, key: str, value: Any) -> dict[str, Any]:  # noqa: ANN
     raise UnsupportedComposeError(msg)
 
 
-def _merge(base: dict[str, Any], local: dict[str, Any], name: str) -> dict[str, Any]:
-    """Merge `local` onto `base` per key category: mapping-merge, sequence-concat, else override."""
+def _merge(base: dict[str, Any], local: dict[str, Any], name: str, base_name: str) -> dict[str, Any]:
+    """Merge `local` onto `base` per key category: mapping-merge, sequence-concat, else override.
+
+    `name` is the extending service, `base_name` the one it extends: each side's
+    failures are reported against the service the offending value actually
+    belongs to.
+    """
     merged: dict[str, Any] = dict(base)
     for key, local_val in local.items():
         spec = SERVICE_KEYS.get(key)
         if key in base and spec is not None and spec.merge is not None:
+            # A merge must never *widen* what the gate accepts. `resolve_extends`
+            # runs ahead of `validate()`, so a normalizing merge (list -> mapping)
+            # can launder a form the key does not have into one it does:
+            # `ulimits: ["nofile=2"]` is refused standalone, but coercing it here
+            # would produce a valid-looking `{"nofile": "2"}` that then sails
+            # through the gate. Checking each side with the key's own validator
+            # first derives the merge's accepted forms from the gate's, so the
+            # two cannot drift apart.
+            spec.validate(base_name, key, base[key])
+            spec.validate(name, key, local_val)
             merged[key] = spec.merge(name, key, base[key], local_val)
         elif key in base and key in _STRUCTURAL_MERGE_KEYS:
-            merged[key] = {**_as_mapping(name, key, base[key]), **_as_mapping(name, key, local_val)}
+            merged[key] = {**_as_mapping(base_name, key, base[key]), **_as_mapping(name, key, local_val)}
         elif key in base and key in _STRUCTURAL_CONCAT_KEYS:
-            merged[key] = concat_list(name, key, base[key], local_val)
+            merged[key] = _as_concat_list(base_name, key, base[key]) + _as_concat_list(name, key, local_val)
         else:
             merged[key] = local_val
     return merged
@@ -115,7 +172,7 @@ def resolve(name: str) -> Any:  # noqa: ANN401 - Compose values are untyped
         if not isinstance(base, dict):
             resolved[name] = local
             return local
-        merged = _merge(base, local, name)
+        merged = _merge(base, local, name, base_name)
         resolved[name] = merged
         return merged
 

planning/changes/2026-07-14.05-extends-merge-policy.md

@@ -0,0 +1,126 @@
+---
+summary: An extends merge may normalize a value only through a form that key actually has, derived from the key's own validator, so a merge can no longer launder a shape the gate refuses standalone (ulimits/cap_add/volumes scalars and lists) nor refuse one it accepts (list-form extra_hosts).
+---
+
+# Design: Derive the extends merge policy from each key's validator
+
+## Summary
+
+`resolve_extends` runs **ahead of** `validate()`. A merge that *normalizes* a
+value (list → mapping, scalar → list) therefore decides what the gate will
+subsequently see — and if it normalizes a form the key does not actually have,
+it turns an invalid document into a valid one. That is what it was doing.
+
+One rule now governs every mergeable key: **a merge may normalize a value only
+through a form that key actually has.** For registry keys the accepted forms are
+derived from the key's own validator, so those cannot drift from the gate by
+construction. Structural keys have no `KeySpec`, so theirs stay declared in
+`extends.py` — kept honest by a test that asserts the invariant for *every*
+mergeable key and fails if a new one is added without a case.
+
+## Motivation
+
+The merge policy had drifted out of step with the gate in *both* directions.
+
+**It laundered invalid input into valid** (the serious half). `ulimits` has no
+list form in Compose, and the gate refuses one:
+
+```
+$ validate({"app": {"ulimits": ["nofile=2"]}})     -> 'ulimits' must be a mapping
+```
+
+But arriving through `extends`, `_merge_map` coerced that list into
+`{"nofile": "2"}` *before* the gate ran, and the document sailed through. The
+same held for every list-only key given a bare scalar — `cap_add`, `devices`,
+`security_opt`, `group_add`, `volumes`, `secrets`, `configs` all refuse a scalar
+standalone, and all silently accepted one via `extends`.
+
+**And it refused valid input.** `extra_hosts` *does* have a list form
+(`- "host:1.2.3.4"`), accepted standalone — but a merge rejected it as an
+"incompatible form".
+
+The asymmetry was not cosmetic: it was two bugs pointing in opposite directions,
+both caused by the policy being an enumeration maintained by hand instead of
+being derived from what each key actually accepts.
+
+## Design
+
+**Registry keys** (`SERVICE_KEYS`): `_merge` runs `spec.validate` on *both* sides
+before `spec.merge`. The merge's accepted forms are therefore exactly the gate's,
+by construction. `ulimits: [...]` and `cap_add: "SCALAR"` now raise — with the
+validator's own message (`'cap_add' must be a list`), which is also the message
+the same value produces standalone.
+
+**Structural keys** (no `KeySpec`): the two normalizers are narrowed to the forms
+Compose actually defines.
+
+- `_as_concat_list` normalizes `scalar → [scalar]` only for `tmpfs` and
+  `env_file` — the only concat keys with a bare-string form, and the only two the
+  gate accepts a scalar for. `volumes`/`secrets`/`configs` must stay lists.
+- `_as_mapping` accepts list form for `environment`, `extra_hosts`, and
+  `depends_on` — each through the normalizer that key's list form actually needs.
+  `healthcheck` has no list form, so a list is still refused.
+
+`extra_hosts` gets a real normalizer: its list form is `host:ip`, **colon**-
+separated, so routing it through `pairs_to_mapping` (which splits on `=`) would
+have silently produced a single `{'host:ip': None}` key. It splits on the *first*
+colon only, so an IPv6 address survives (`myhost:::1` → `{'myhost': '::1'}`).
+
+No structural-key registry — `decisions/2026-07-12` stands. This derives policy
+from validators that already exist; it does not build a dispatch table.
+
+## The invariant, pinned
+
+A parametrized test asserts, for **every** mergeable key, that a form the gate
+refuses standalone is also refused through `extends`, using the shape that
+actually laundered (a bare scalar for the list-only keys, a list for the
+mapping-only ones) rather than merely any invalid value. A companion test
+asserts the table covers every mergeable key, so a new key cannot be added
+without a case — the drift this change exists to end.
+
+Run against the pre-fix `extends.py` it goes red on nine keys (`cap_add`,
+`cap_drop`, `configs`, `devices`, `group_add`, `secrets`, `security_opt`,
+`ulimits`, `volumes`), which is the laundering it now prevents.
+
+## Non-goals
+
+- Not validating the whole service pre-merge (the heavier structural option).
+  Deriving per-key from the validator closes the class with a much smaller
+  change, and the invariant test guards the seam.
+- Not changing what the gate itself accepts. Every shape valid standalone before
+  this change is still valid; every shape invalid standalone is now *also*
+  invalid through `extends`.
+
+## Behavior change
+
+Three previously-accepted merges now raise, all of which produced documents the
+gate would have refused standalone:
+
+- `ulimits` in list form on a merged side
+- any list-only key (`cap_add`, `devices`, `security_opt`, `group_add`,
+  `volumes`, `secrets`, `configs`) given a bare scalar on a merged side
+- (unchanged, still refused) `healthcheck` in list form
+
+One previously-refused merge now works: list-form `extra_hosts`.
+
+Registry-key merge errors now come from the key's own validator (`'cap_add'
+must be a list`) rather than the vaguer `cannot merge 'cap_add' across
+incompatible forms`, matching what the same value yields standalone. Structural
+keys keep the `cannot merge ...` message.
+
+Merge errors also now name the service the offending value belongs to: a
+malformed value in the *base* is reported against the base, not against the
+service extending it.
+
+## Testing
+
+`just test-ci` at 100%. The invariant test above is the centrepiece; new tests
+cover the `extra_hosts` list form (including IPv6 colon-safety and a
+missing-colon refusal) and the scalar-form narrowing.
+
+## Risk
+
+- **A user relying on the laundering** — e.g. a base with `cap_add: "NET_ADMIN"`
+  (scalar) — now gets an error. This is the intended correction: that document
+  was already invalid without `extends`, and the tool was accepting it only by
+  accident of merge order. The error names the key and the required form.