D3: serve images and videos through the Python reference worker

README.md

@@ -164,7 +164,7 @@ Python screenshot worker, Valen-Preview-0923 has a Python reference worker for
 text and one-image state, CLM has a stub-encoder contract recipe, and OmniJev-4B
 v1.1 has its checkpoint export and CPU contract, with no worker yet. D3
 (Decision 3.0) has a Python reference worker that runs the checkpoint's own
-pinned runtime for text state:
+pinned runtime for text, image and video state:
 
 | Model | Status |
 | --- | --- |
@@ -178,7 +178,7 @@ pinned runtime for text state:
 | CLM-v0.1-8B | [External worker with a CPU stub encoder](recipe/clm/README.md); contract checks only, real Qwen3-8B decisions unverified by this recipe |
 | LFM2.5-350M | [Transformers worker](recipe/lfm2/README.md); text state and independent choice questions |
 | OmniJev-4B v1.1 | [Checkpoint export and CPU contract](recipe/omnijev/README.md) only; the native worker is in progress ([#114](https://github.com/ThinkFlowLab/system1-omni/issues/114)) |
-| D3 (Decision 3.0) | [Python reference worker](recipe/d3/reference.md) on the checkpoint's pinned runtime; `choice`/`noul`/`score` text state, media refused until stage 2; author-run on A100 (evidence in the integration PR) |
+| D3 (Decision 3.0) | [Python reference worker](recipe/d3/reference.md) on the checkpoint's pinned runtime; `choice`/`noul`/`score` over text, image and video state; author-run on A100 for text and media serving (see the integration and media PRs) |
 
 [Supported models and hardware](docs/supported-models.md) lists the devices
 and where each worker has been run.

docs/supported-models.md

@@ -23,7 +23,7 @@ Models that are being added are also tracked in issues labeled [new model](https
 | LFM2.5-350M | [Transformers choice worker](../recipe/lfm2/README.md) | Tiny-model regression tests; full checkpoint unverified | L40S FP16 [recorded execution checks](../recipe/lfm2/RESULTS.md) | Unverified | Python 3.12, [pinned dependencies](../src/models/lfm2/requirements.lock.txt), explicit candidate batch size |
 | CLM-v0.1-8B | [Native Rust engine](../src/models/clm/README.md), `clm-run`, in front of a Qwen3-8B `/v1/embeddings` server | Engine-only agreement **4.5e-06** on identical vectors ([#29](https://github.com/ThinkFlowLab/system1-omni/pull/29), [validation](../recipe/clm/native/VALIDATION.md)); contract checks documented ([#23](https://github.com/ThinkFlowLab/system1-omni/pull/23)) | **Validated on compute capability 8.9** ([#29](https://github.com/ThinkFlowLab/system1-omni/pull/29), [validation](../recipe/clm/native/VALIDATION.md)) | Unverified | Rust to build and a separate Qwen3-8B `/v1/embeddings` server for the encoder; the engine itself has no device-specific code |
 | OmniJev-4B v1.1 | None yet: [checkpoint export and CPU contract](../recipe/omnijev/README.md); the native worker is in progress ([#114](https://github.com/ThinkFlowLab/system1-omni/issues/114)) | Request contract, preparation, heads and finishing checked against the reference ([#120](https://github.com/ThinkFlowLab/system1-omni/pull/120)); no inference | Not yet | Not supported | Python 3.10 reference environment for the export and fixtures |
-| D3 (Decision 3.0) | [Python reference worker](../recipe/d3/reference.md) running the checkpoint's pinned runtime | Unverified | Author-run on NVIDIA A100 (sm_80): three-kind serving smoke with byte-identical frontend forwarding and reference parity (exact decisions, max delta 0.0, 348 == 348 tokens); evidence in the integration PR. Owner: @MrDongsls. | Unverified | Python 3.12, the [pinned runtime requirements](../recipe/d3/requirements.txt), the [pinned package](https://huggingface.co/vllm-sr/d3) (~52 GB) |
+| D3 (Decision 3.0) | [Python reference worker](../recipe/d3/reference.md) on the checkpoint's pinned runtime — text, image and video state over `choice`/`noul`/`score`, with the deliberate transport differences documented in the recipe; the native worker is in progress ([#136](https://github.com/ThinkFlowLab/system1-omni/issues/136)) | Unverified | Author-run on NVIDIA A100 (sm_80): serving smoke and reference parity for text and media (exact decisions, max delta 0.0); evidence in the integration and media PRs. Owner: @MrDongsls. | Unverified | Python 3.12, the [pinned runtime requirements](../recipe/d3/requirements.txt), the [pinned package](https://huggingface.co/vllm-sr/d3) (~52 GB); [media budgets and the 64 MiB total request cap](../recipe/d3/reference.md) |
 
 - **Validated:** covered by the recipe on `main` or by the checks in the linked merged pull request.
 - **Author-run:** recorded by the recipe author on the stated hardware and not independently reproduced; the row names its owner.
@@ -32,8 +32,9 @@ Models that are being added are also tracked in issues labeled [new model](https
 
 The Cua-S1 workers answer `choice` questions only.
 LAYA's English worker, Open-Jev and CLM support `choice`, `score`, and `noul` text questions.
-D3's reference worker serves the same three kinds for text state; its media
-inputs stay refused for now and will be integrated soon.
+D3's reference worker serves the same three kinds for text, image and video
+state (inline base64 media under the pinned runtime's per-item budgets);
+its media serving evidence is author-run on the A100 above.
 CLM has a row per path: the merged `clm-serve` recipe exercises those answer shapes with stub
 embeddings and says nothing about decision quality, while the native engine's CUDA row is the
 path through a real Qwen3-8B encoder. Its engine-only figure comes from the same comparison run

recipe/README.md

@@ -26,8 +26,9 @@ For a first real decision, follow the [complete CPU walkthrough](../docs/getting
 - [CLM behind the frontend](clm/README.md): run CLM's own server behind the frontend on
   CPU with a stub encoder, and what the response comparison has to allow for.
 - [D3 reference worker](d3/reference.md): download the pinned vllm-sr/d3 package,
-  verify its runtime manifest, and serve `choice`/`noul`/`score` text decisions
-  through the Python worker and the Rust frontend (serving and parity validated on A100).
+  verify its runtime manifest, and serve `choice`/`noul`/`score` decisions over
+  text, image and video state through the Python worker and the Rust frontend
+  (text serving and parity validated on A100; media budgets and caps in the recipe).
 
 - [LFM2.5-350M choice worker](lfm2/README.md): score text decisions with independent
   questions, hybrid cache branching and configurable candidate batches.

recipe/d3/compare_reference.py

@@ -5,15 +5,18 @@
 the comparison isolates this repository's admission and orchestration from
 the checkpoint's own code:
 
-- reference: the decide() body of the upstream ``d3_server.py`` (prepare,
-  reject over-limit and invalid questions with 422, run, respond), driven
-  directly on the loaded runtime;
+- reference: the ``system_one`` body of the upstream ``d3_server.py``
+  (decode_images/decode_videos with their empty short-circuits and
+  capability gates, then the decide body: prepare with the media-aware
+  branches, reject over-limit and invalid questions with 422, run,
+  respond), driven directly on the loaded runtime;
 - worker: the production ``frontend.d3.decide`` pipeline (protocol envelope,
-  prepared-error admission, run, respond, response finishing).
+  decode stage, prepared-error admission, run, respond, response finishing).
 
 Both paths execute the same verified ``d3_runtime``, so the criteria are
 exact decision agreement (including embedded ``invalid_model_output``
-shapes), equal token accounting, and a probability tolerance of 1e-6
+shapes and agreed 422 rejections, media decode errors among them), equal
+token accounting, and a probability tolerance of 1e-6
 (``--tolerance``). ``--export`` additionally freezes the oracle fixture —
 token ids, probabilities and answers per question — for the future native
 worker, and writes it only after a passing, non-rejected comparison.
@@ -53,14 +56,57 @@ def parse_args(argv=None):
     return parser.parse_args(argv)
 
 
+def _decode_images(model, images):
+    """Upstream d3_server.py decode_images, verbatim."""
+
+    if not isinstance(images, list):
+        raise ValueError("images must be a list of base64 data URLs")
+    if not images:
+        return []
+    if getattr(model, "image_unavailable", "unknown") is not None:
+        raise ValueError("This model reads text only; images are not supported.")
+    return model.load_images(images, strict=True)
+
+
+def _decode_videos(model, videos):
+    """Upstream d3_server.py decode_videos, verbatim."""
+
+    if not isinstance(videos, list):
+        raise ValueError("videos must be a list of base64 data URLs")
+    if not videos:
+        return []
+    if getattr(model, "video_unavailable", "unknown") is not None:
+        raise ValueError("This model does not read videos.")
+    return model.load_videos(videos, strict=True)
+
+
+def decode_request_media(model, body):
+    """The media half of upstream system_one: images then videos."""
+
+    images = _decode_images(model, body["images"]) if body.get("images") is not None else []
+    videos = _decode_videos(model, body["videos"]) if body.get("videos") is not None else []
+    return images, videos
+
+
 def reference_decide(model, body):
-    """The upstream d3_server.py decide() body, without FastAPI.
+    """The upstream d3_server.py system_one body, without FastAPI.
 
     Returns the 200 response dict, or a ("422", detail) tuple when the
-    upstream server would have rejected the request before execution.
+    upstream server would have rejected the request before execution —
+    including media rejections from the decode stage, which upstream maps
+    to a plain-detail 422.
     """
 
-    prepared = model.prepare(body.get("state"), body.get("questions"))
+    try:
+        images, videos = decode_request_media(model, body)
+        if videos:
+            prepared = model.prepare(body.get("state"), body.get("questions"), images, videos)
+        elif images:
+            prepared = model.prepare(body.get("state"), body.get("questions"), images)
+        else:
+            prepared = model.prepare(body.get("state"), body.get("questions"))
+    except ValueError as exc:
+        return ("422", str(exc))
     over = [e for e in prepared.errors.values() if e["error"] == "max_length_exceeded"]
     if over:
         return ("422", over[0]["message"])
@@ -113,7 +159,7 @@ def compare_case(name, worker_response, reference, tolerance=PROBABILITY_TOLERAN
     Returns (report lines, passed).
     """
 
-    lines = [f"case: {name}", "reference path: d3_server decide()", "worker model: d3"]
+    lines = [f"case: {name}", "reference path: d3_server system_one()", "worker model: d3"]
     problems = []
     deltas: list[float] = []
     if isinstance(reference, tuple):
@@ -123,9 +169,16 @@ def compare_case(name, worker_response, reference, tolerance=PROBABILITY_TOLERAN
         else:
             detail = reference[1]
             body = rejection.get("body")
-            agreed = rejection.get("status") == 422 and body == (
-                detail if isinstance(detail, dict) else {"detail": detail}
-            )
+            if isinstance(detail, dict):
+                agreed = rejection.get("status") == 422 and body == detail
+            else:
+                # The worker deliberately wraps prepare-stage rejections in a
+                # fixed prefix (src/frontend/d3.py decide); upstream system_one
+                # surfaces the raw runtime message. Both forms agree here.
+                agreed = rejection.get("status") == 422 and body in (
+                    {"detail": detail},
+                    {"detail": "request rejected by the pinned runtime: " + detail},
+                )
             lines.append(f"worker rejection: {rejection.get('status')} {body}")
             lines.append("rejection agreement: " + ("PASS" if agreed else "FAIL"))
             return lines, bool(agreed)
@@ -220,7 +273,10 @@ def main(argv=None):
         lines, ok = compare_case(Path(args.fixture).name, worker_response, reference, args.tolerance)
         print("\n".join(lines))
         if args.export and export_allowed(ok, worker_response, reference):
-            prepared = executor.prepare(body.get("state"), dict(body["questions"]))
+            images, videos = decode_request_media(executor.model, body)
+            prepared = executor.prepare(
+                body.get("state"), dict(body["questions"]), images, videos
+            )
             probabilities, _ = executor.run(prepared)
             export_oracle(body, prepared, probabilities, worker_response, args.export)
             print(f"oracle exported: {args.export}")

recipe/d3/example-request-media.json

@@ -0,0 +1,26 @@
+{
+  "model": "d3",
+  "state": "The attached screenshot shows the customer's order history: one order charged twice on the same card.",
+  "questions": {
+    "refund": {
+      "type": "choice",
+      "instructions": "Decide the refund action.",
+      "criteria": {
+        "refund": "Refund the duplicate charge now",
+        "wait": "Wait for manual review"
+      }
+    },
+    "severity": {
+      "type": "score",
+      "instructions": "Rate the severity of the billing error.",
+      "criteria": [
+        "minor",
+        "moderate",
+        "major"
+      ]
+    }
+  },
+  "images": [
+    "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAIAAAAlC+aJAAABTElEQVR4nO3ayxHCMAwEUIWhJVIBZVMBqSd37iAnkfXZNeM9Mhz8sB1sKcu+7zJybugBeDMB6EwAOn8N2J5r2Ti6s7T+BwhH/3i9fz/UZ4Bw9K3oANXKmfv1r6JUx8vB8BTiXFe2xyih4RzwtXLYDJdmgNlwdQnRGgx7gNNg28SEBvNhjs3QcxqlMnQep3kM/fcBEoPrQsNg8N7I4IaAKyXWEHMnBhrCLvUoQ2RVAmIILqvUG+LrQsWGlMJWpSGrMldmSCwt1hhya6MFhvTibrahojqdaigqr+cZ6voDSYbSBkeGobpDE24AtJhiDZgeWaAB1uSLMiC7lCEGcJvVb8D3iZ0GPEB8BgqAOAwsAOk1EAGky8AFELuBDiDGjjojQCyG81cN4E2k45DOgBr1pxwJoK6r5gtPVNmea2tXjAE4yEhLSM0EoDMB6HwA6F2Vhz1WcLcAAAAASUVORK5CYII="
+  ]
+}