jev_vl: process inline images with the shared native vision backend

.github/workflows/ci.yml

@@ -42,6 +42,7 @@ jobs:
       - run: python -m pip install -r benchmarks/requirements.txt
       - run: python benchmarks/bench.py validate benchmarks/smoke.jsonl
       - run: python -m unittest discover -s tests/benchmarks -p 'test_*.py' -v
+      - run: python -m unittest discover -s tests/jev_vl -p 'test_export_vision.py' -v
       - name: Decider validation helpers
         run: python -m unittest discover -s tests/decider -p 'test_*.py' -v
 

README.md

@@ -175,7 +175,7 @@ pinned runtime for text state:
 | Valen-Preview-0923 | [Python reference worker](recipe/valen/reference.md); text state or one inline PNG/JPEG image, `choice`; author-run on RTX 5060 Laptop GPU (sm_120, WSL2), owner: community |
 | Open-Jev-27B-v1.1 | [Native Rust/CUDA worker](recipe/open_jev/native.md); eager independent text candidates; [H200 validation](recipe/open_jev/validation.md) |
 | Open-Jev-9B | The same [native Rust/CUDA worker](recipe/open_jev/native.md); eager independent text candidates; [reference comparison on sm_89](recipe/open_jev/validation-9b.md) |
-| autotrust/JEV-27B-VL | [Experimental Rust/CUDA worker](recipe/jev_vl/README.md); single-question text and offline-preencoded image inputs; [bounded H800 validation and limits](recipe/jev_vl/validation.md) |
+| autotrust/JEV-27B-VL | [Experimental Rust/CUDA worker](recipe/jev_vl/README.md); single-question text, online inline PNG/JPEG and offline-preencoded image inputs; [validation scope and limits](recipe/jev_vl/validation.md) |
 | CLM-v0.1-8B | [External worker with a CPU stub encoder](recipe/clm/README.md); contract checks only, real Qwen3-8B decisions unverified by this recipe |
 | LFM2.5-350M | [Transformers worker](recipe/lfm2/README.md); text state and independent choice questions |
 | OmniJev-4B v1.1 | [Checkpoint export and CPU contract](recipe/omnijev/README.md) only; the native worker is in progress ([#114](https://github.com/ThinkFlowLab/system1-omni/issues/114)) |

docs/architecture.md

@@ -73,10 +73,11 @@ warmup before readiness remain model-specific and unchanged.
 
 The experimental [JEV-VL worker](../recipe/jev_vl/README.md) also uses the shared
 Qwen executor. It admits one official single-decision request at a time, reads
-selected LM-head rows, and can retain image-prefix state across requests. Image
-features must be prepared offline. Its `{kind, state, question, options}` contract
-is distinct from the existing `{model, state, questions}` envelope; frontend
-transport alone does not adapt those schemas.
+selected LM-head rows, and can retain image-prefix state across requests. Images
+are either prepared offline by Transformers or supplied inline and encoded
+inside the worker by the shared native vision backend. Its `{kind, state,
+question, options}` contract is distinct from the existing `{model, state,
+questions}` envelope; frontend transport alone does not adapt those schemas.
 
 ## Layer ownership and implementation language
 

recipe/cua_s1/native_vision_graph.md

@@ -5,7 +5,8 @@ encoder, including all 24 blocks, unmerged FP32 LoRA branches and the merger.
 Set `CUA_S1_VISION_GRAPH=1` before loading the worker to enable it. Leave it
 unset or set it to `0` for eager execution. This switch is independent of
 `CUA_S1_GRAPH`, which controls text and multimodal language execution. Enable
-each switch separately; keep both unset for eager execution.
+each switch separately; keep both unset for eager execution. Enabling vision
+Graph alone does not enable language Graph replay.
 
 Both modes use the shared Qwen vision executor and retain at most four exact
 [T,H,W] scratch/Graph pairs in FIFO order. Equal patch counts with different

recipe/jev_vl/README.md

@@ -1,7 +1,8 @@
 # JEV-27B-VL experimental native recipe
 
 This recipe serves `autotrust/JEV-27B-VL` System-1 decisions with Rust/CUDA.
-Text runs natively; images must first be encoded offline by Transformers.
+Text runs natively. Images can use the shared native vision backend or assets
+encoded offline by Transformers.
 The [model contract](../../src/models/jev_vl/README.md) explains the single-question
 API, verbalizer head and cache ownership. The reviewed candidate passed a
 [bounded H800 validation](validation.md#historical-h800-validation);
@@ -52,7 +53,8 @@ JEV_VL_MODEL=weights/jev-vl-merged \
 `JEV_VL_HOST` defaults to `127.0.0.1`, `JEV_VL_PORT` to `8001`, and the CUDA
 library defaults to the file beside the executable. The worker uses visible
 CUDA device 0. `/health` becomes available after model loading and a successful
-text warmup; that health check does not validate image assets.
+text warmup. With `JEV_VL_VISION`, readiness also requires a successful inline-image
+warmup; it does not validate every user image or offline-preencoded asset.
 
 In separate terminals, start the frontend and send the same request directly and
 through it:
@@ -81,10 +83,66 @@ checks. The frontend routes `/health` and `/v1/systemone`; it does not proxy
 generation routes such as `/v1/chat/completions`. An unknown route returns the
 frontend's own 404 rather than this worker's error envelope.
 
-## Prepare images before serving
+## Online images
 
-The worker does not decode images, fetch URLs or invoke a vision tower on cache
-miss. First prepare a JSONL manifest with one object per line containing a
+Export the vision tower separately from the same pinned source checkpoint.
+The CPU exporter preserves BF16 vision weights and pins the processor and source
+files. It does not rewrite the language export:
+
+```sh
+CUDA_VISIBLE_DEVICES='' .venv-jev-vl/bin/python recipe/jev_vl/export_vision.py \
+  --model weights/JEV-27B-VL --out weights/jev-vl-vision
+JEV_VL_MODEL=weights/jev-vl-merged JEV_VL_VISION=weights/jev-vl-vision \
+  JEV_VL_CUDA_LIB=$PWD/target/release/libqwen3_5_cuda.so \
+  JEV_VL_CACHE=1 CUA_S1_VISION_GRAPH=0 target/release/omni-jev-vl-native
+```
+
+With `JEV_VL_VISION` set, startup runs an online vision warmup after the text
+warmup: one full inline-image decision, a 1×1 PNG through decode, preprocessing
+and the vision forward. The worker does not bind its port unless that warmup
+succeeds, so in online mode `/health` implies the inline image path executed
+end to end once at startup. It does not validate future user images. New image
+inputs are decoded and checked when prepared; cache hits may reuse validated
+assets or retained image-prefix state.
+
+Online mode accepts single-frame inline PNG/JPEG data URIs. Each image is bounded
+to 4 MiB of decoded file bytes, 2048 pixels per side, 1,048,576 source pixels,
+and 4608 vision patches. The full HTTP body is limited to 8 MiB, including base64
+and text, so at least one maximum-size image fits; requests beyond the body
+limit are rejected with status 413 in the same error envelope. Remote URLs,
+animations and video are unsupported. Oversized inputs are rejected; resource
+limits do not change the model's resize policy.
+
+No per-request image-count limit applies beyond the body and token budgets.
+A minimum-size image uses 1.5 MiB of FP32 processed pixels and produces
+0.625 MiB of BF16 embeddings. Pixel buffers are released as image inputs are
+consumed during vision resolution; embeddings remain for language execution.
+Cache budgets exclude in-flight inputs. Hundreds of tiny images can therefore
+require hundreds of MiB of host memory, plus temporary copies, weights, scratch
+and caches. These array sizes are not a measured peak-memory bound.
+
+The shared vision backend also reads `CUA_S1_VISION_GRAPH` and
+`CUA_S1_GRAPH_TRACE`, including when called by JEV. They enable capture and
+capture diagnostics respectively only when set to `1`; both default to off.
+Set `CUA_S1_VISION_GRAPH=0` explicitly as above to prevent an inherited Cua
+setting from changing the JEV execution path. Current JEV validation uses
+eager vision and does not cover the opt-in graph path.
+
+`JEV_VL_VISION` and `JEV_VL_IMGCACHE` are mutually exclusive. Online mode also
+ignores the default `imgcache/` directory. The source and processor stay fixed
+for the worker lifetime. A new image runs decode, preprocessing and vision
+inside the request; repeated images reuse the bounded L2 cache. Vision and
+language execution share the existing serial scheduler. Rebuild the CUDA library
+to include the shared 27B vision entry points.
+
+Historical prepared-image measurements below do not validate this online path
+or include its vision cost. Compare fresh images and repeated-image questions
+separately against the official model before reporting online performance.
+
+## Prepared images
+
+Without `JEV_VL_VISION`, the worker reads prepared assets on cache miss.
+First prepare a JSONL manifest with one object per line containing a
 `request` whose `state` includes `{"image":"data:image/png;base64,..."}`. The
 preencoder accepts base64 data URIs; a fresh URL or changed image needs a new
 asset. It runs on a GPU and should finish before starting the language worker
@@ -111,16 +169,20 @@ not an end-to-end live screenshot service.
 
 | Variable | Default | Meaning |
 | --- | --- | --- |
-| `JEV_VL_CACHE` | `1` | Master enable; `0` uses a full language forward and rereads prepared image assets. |
+| `JEV_VL_CACHE` | `1` | Master enable; `0` uses a full language forward and recomputes online vision or rereads prepared assets. |
 | `JEV_VL_L1`, `JEV_VL_L2`, `JEV_VL_L3` | `1` | Processor records, parsed image assets, and language-prefix state. |
 | `JEV_VL_L1_MAX` | `256` | Maximum resident processor records. |
 | `JEV_VL_L2_BYTES` | `1073741824` | Parsed image-asset cache budget. |
 | `JEV_VL_L3_BYTES` | `2147483648` | Cached language-prefix device-state budget. |
 
 Cache budgets do not include weights, model scratch, in-flight request inputs or
 total process memory. `GET /v1/cache/stats` exposes counters and resident cache
-accounting. Use those counters to assess L2 hits; `x-jev-cache` does not report
-an L2 hit field. `POST /v1/cache/reset` resets counters only; restart the worker for
+accounting. `l2_hit` and `l2_miss` count lookups, not unique images or requests.
+Cold online images are checked during preparation and again after queueing;
+prepared mode normally checks once. Retained-prefix continuations may skip L2
+entirely. Compare these counters within the same execution path rather than as
+cross-mode per-image hit rates. `x-jev-cache` does not report an L2 hit field.
+`POST /v1/cache/reset` resets counters only; restart the worker for
 a cold cache. Request preparation may overlap, while GPU execution stays serial.
 Use cache modes as experimental controls within the documented validation scope.
 

recipe/jev_vl/export_vision.py

@@ -0,0 +1,96 @@
+"""Export the pinned JEV vision tower for the shared Rust/CUDA backend (CPU only)."""
+
+import argparse
+import hashlib
+import json
+import shutil
+from pathlib import Path
+
+
+def sha256(path):
+    digest = hashlib.sha256()
+    with path.open("rb") as stream:
+        for block in iter(lambda: stream.read(4 << 20), b""):
+            digest.update(block)
+    return digest.hexdigest()
+
+
+def export(model, out):
+    if out.exists():
+        raise ValueError("output already exists; choose a new export directory")
+    config = json.loads((model / "config.json").read_text())
+    if (config.get("model_type") != "qwen3_5"
+            or config.get("vision_config", {}).get("hidden_size") != 1152
+            or config.get("vision_config", {}).get("out_hidden_size") != 5120
+            or config.get("text_config", {}).get("hidden_size") != 5120):
+        raise ValueError("expected the JEV-27B-VL Qwen3.8 backbone")
+    processor = json.loads((model / "preprocessor_config.json").read_text())
+    expected_processor = {
+        "size": {"longest_edge": 16777216, "shortest_edge": 65536},
+        "patch_size": 16, "temporal_patch_size": 2, "merge_size": 2,
+        "image_mean": [0.5, 0.5, 0.5], "image_std": [0.5, 0.5, 0.5],
+        "processor_class": "Qwen3VLProcessor",
+        "image_processor_type": "Qwen2VLImageProcessorFast",
+    }
+    if processor != expected_processor:
+        raise ValueError("unsupported image processor; verify resize and normalization first")
+
+    import torch
+    from safetensors import safe_open
+    from safetensors.torch import save_file
+
+    adapter = model / "adapter_vllm"
+    with safe_open(adapter / "adapter_model.safetensors", framework="pt") as weights:
+        for name in weights.keys():
+            if not name.startswith(("base_model.model.model.language_model.",
+                                    "base_model.model.lm_head.")):
+                raise ValueError(f"unsupported non-language adapter tensor: {name}")
+
+    index = json.loads((model / "model.safetensors.index.json").read_text())["weight_map"]
+    names = {name for name in index if name.startswith("model.visual.")}
+    if not names:
+        raise ValueError("checkpoint contains no vision tensors")
+    tensors = {}
+    for shard in sorted({index[name] for name in names}):
+        path = (model / shard).resolve()
+        if not path.is_relative_to(model.resolve()):
+            raise ValueError("checkpoint shard escapes model directory")
+        with safe_open(path, framework="pt") as weights:
+            for name in sorted(names):
+                if index[name] != shard:
+                    continue
+                tensor = weights.get_tensor(name)
+                if tensor.dtype != torch.bfloat16 or not torch.isfinite(tensor).all():
+                    raise ValueError(f"expected finite BF16 vision weights: {name}")
+                tensors[name] = tensor.contiguous()
+
+    out.mkdir(parents=True)
+    save_file(tensors, out / "vision.safetensors")
+    for name in ("config.json", "preprocessor_config.json"):
+        shutil.copyfile(model / name, out / name)
+    manifest = {
+        "format": "jev-vl-vision/1",
+        "model_id": "autotrust/JEV-27B-VL",
+        "pins": {
+            "model_index_sha256": sha256(model / "model.safetensors.index.json"),
+            "adapter_config_sha256": sha256(adapter / "adapter_config.json"),
+            "adapter_model_sha256": sha256(adapter / "adapter_model.safetensors"),
+        },
+        "files": {name: sha256(out / name) for name in
+                  ("config.json", "preprocessor_config.json", "vision.safetensors")},
+    }
+    # The worker requires this manifest, so a partial export is never loadable.
+    (out / "jev_vl_vision.json").write_text(json.dumps(manifest, indent=2) + "\n")
+    print(f"exported {len(tensors)} vision tensors -> {out}")
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--model", type=Path, required=True)
+    parser.add_argument("--out", type=Path, required=True)
+    args = parser.parse_args()
+    export(args.model, args.out)
+
+
+if __name__ == "__main__":
+    main()

recipe/jev_vl/preencode.py

@@ -5,7 +5,7 @@
 ``<out>/<sha256(url)>/{emb.safetensors,grid.json}`` with the adapted rows
 (``model.model.visual(pixel_values, grid_thw).pooler_output``, shape
 [n = prod(grid)/merge^2, 5120], bfloat16) plus the patch grid. The L2 cache holds
-these assets; the native worker never decodes images.
+these assets in prepared-image mode. Online mode uses its own native vision tower.
 """
 import argparse
 import hashlib