Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ v1.1 has its checkpoint export and CPU contract, with no worker yet:
| --- | --- |
| LAYA | [External worker](recipe/laya/README.md); [Python worker on Apple Silicon (MPS) and CPU](recipe/laya/apple-silicon.md); [CPU checkpoint reader](src/models/laya/README.md); [native Rust/CUDA worker on Hopper](recipe/laya/native/README.md) |
| Cua-S1 4B 0.2 (`text` adapter) | [Python worker](recipe/cua_s1/text.md); [native worker](recipe/cua_s1/native.md), CUDA, run on sm_89 |
| Cua-S1 4B 0.2 (`multimodal` adapter) | [Python CUDA worker](src/frontend/cua_s1.py); one PNG/JPEG screenshot, `choice`; native screenshot execution remains in progress |
| Cua-S1 4B 0.2 (`multimodal` adapter) | [Python CUDA reference](src/frontend/cua_s1.py) and [native Rust/CUDA screenshot worker](recipe/cua_s1/native_screenshot.md); one PNG/JPEG image, `choice`; scoped RTX 4090 validation |
| Valen-Preview-0923 | [Python reference worker](recipe/valen/reference.md); text state or one inline PNG/JPEG image, `choice`; author-run on RTX 5060 Laptop GPU (sm_120, WSL2), owner: community |
| Open-Jev-27B-v1.1 | [Native Rust/CUDA worker](recipe/open_jev/native.md); eager independent text candidates; [H200 validation](recipe/open_jev/validation.md) |
| Open-Jev-9B | The same [native Rust/CUDA worker](recipe/open_jev/native.md); eager independent text candidates; [reference comparison on sm_89](recipe/open_jev/validation-9b.md) |
Expand Down
3 changes: 2 additions & 1 deletion docs/supported-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,11 @@ Models that are being added are also tracked in issues labeled [new model](https
| Cua-S1 4B 0.2, `text` adapter | [Reference worker](../recipe/cua_s1/text.md) on Transformers and PEFT | Unverified | Validated ([#13](https://github.com/ThinkFlowLab/system1-omni/pull/13)) | Unverified | Python 3.12, the versions in `requirements-text.txt` |
| Cua-S1 4B 0.2, `text` adapter | [Native Rust worker](../recipe/cua_s1/native.md) on the [Qwen3.5 CUDA kernels](../src/backends/cuda/qwen3_5/README.md) | Not supported | Validated on compute capability 8.9 ([#19](https://github.com/ThinkFlowLab/system1-omni/pull/19), [#52](https://github.com/ThinkFlowLab/system1-omni/pull/52)) | Not supported | Compute capability 8.0 or newer, the CUDA toolkit to build, weights merged with `export_text_merged.py` |
| Cua-S1 4B 0.2, `multimodal` adapter | Reference worker on Transformers and PEFT, [`src/frontend/cua_s1.py`](../src/frontend/cua_s1.py); no recipe yet | Not supported | Validated ([#17](https://github.com/ThinkFlowLab/system1-omni/pull/17), [#18](https://github.com/ThinkFlowLab/system1-omni/pull/18)) | Not supported | The state is one PNG or JPEG image; upstream's `weights.lock.json` next to the base weights |
| Cua-S1 4B 0.2, `multimodal` adapter | [Native Rust/CUDA screenshot worker](../recipe/cua_s1/native_screenshot.md) | Preparation checks only; no CPU inference | RTX 4090 / sm_89, native vision and language Graph regressions; [scoped evidence and controls](../recipe/cua_s1/native_vision_graph.md) | Not supported | Verified BF16 language export, original vision checkpoint and FP32 vision adapter, rebuilt ABI 8 CUDA library; one PNG/JPEG and `choice` |
| Valen-Preview-0923 | [Python reference worker](../recipe/valen/reference.md); text state or one inline PNG/JPEG image; `choice` questions | Not supported | Author-run on RTX 5060 Laptop (sm_120, WSL2): one text and one 2×2 PNG request; input and readout parity ≤1e-6 against reference (model outputs not compared). Owner: community. | Not supported | Python 3.12, pinned Valen source/checkpoint, Qwen3.5-2B base, BF16 CUDA |
| Open-Jev-27B-v1.1 | [Native Rust/CUDA worker](../recipe/open_jev/native.md) on the shared Qwen3.5/3.8 executor | Not supported | Validated on H200 (sm_90) for the [74 single-candidate workload](../recipe/open_jev/validation.md) | Not supported | Compute capability 8.0 or newer, CUDA toolkit to build, exported merged weights and trained head |
| Open-Jev-9B | The same [native Rust/CUDA worker](../recipe/open_jev/native.md) | Not supported | Validated on compute capability 8.9 against the reference for [253 requests](../recipe/open_jev/validation-9b.md) | Not supported | Compute capability 8.0 or newer, CUDA toolkit to build, exported merged weights and trained head |
| autotrust/JEV-27B-VL | [Experimental native worker](../recipe/jev_vl/README.md), single-question API | Contract tests only; no CPU inference | H800 frozen-corpus check; [experimental scope and limits](../recipe/jev_vl/validation.md) | Not supported | Pinned merged checkpoint, rebuilt ABI 6 CUDA library; image inputs require offline Transformers preencoding |
| autotrust/JEV-27B-VL | [Experimental native worker](../recipe/jev_vl/README.md), single-question API | Contract tests only; no CPU inference | H800 frozen-corpus check; [experimental scope and limits](../recipe/jev_vl/validation.md) | Not supported | Pinned merged checkpoint, rebuilt ABI 8 CUDA library; image inputs require offline Transformers preencoding |
| CLM-v0.1-8B | [External `clm-serve` recipe](../recipe/clm/README.md) with a CPU stub embeddings server | **Stub-encoder contract checks only** ([#23](https://github.com/ThinkFlowLab/system1-omni/pull/23)); not real Qwen3-8B decisions | Real encoder unverified by the merged recipe | Unverified | Python, upstream CLM and head checkpoint; a real encoder requires a separate embeddings server |
| LFM2.5-350M | [Transformers choice worker](../recipe/lfm2/README.md) | Tiny-model regression tests; full checkpoint unverified | L40S FP16 [recorded execution checks](../recipe/lfm2/RESULTS.md) | Unverified | Python 3.12, [pinned dependencies](../src/models/lfm2/requirements.lock.txt), explicit candidate batch size |
| CLM-v0.1-8B | [Native Rust engine](../src/models/clm/README.md), `clm-run`, in front of a Qwen3-8B `/v1/embeddings` server | Engine-only agreement **4.5e-06** on identical vectors ([#29](https://github.com/ThinkFlowLab/system1-omni/pull/29), [validation](../recipe/clm/native/VALIDATION.md)); contract checks documented ([#23](https://github.com/ThinkFlowLab/system1-omni/pull/23)) | **Validated on compute capability 8.9** ([#29](https://github.com/ThinkFlowLab/system1-omni/pull/29), [validation](../recipe/clm/native/VALIDATION.md)) | Unverified | Rust to build and a separate Qwen3-8B `/v1/embeddings` server for the encoder; the engine itself has no device-specific code |
Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ nav:
- Cua-S1 text worker: recipe/cua_s1/text.md
- Cua-S1 native text worker: recipe/cua_s1/native.md
- Valen reference worker: recipe/valen/reference.md
- Cua-S1 native screenshots: recipe/cua_s1/native_screenshot.md
- Cua-S1 vision CUDA Graph: recipe/cua_s1/native_vision_graph.md
- Open-Jev native text worker: recipe/open_jev/native.md
- Open-Jev-27B H200 validation: recipe/open_jev/validation.md
Expand Down
2 changes: 2 additions & 0 deletions recipe/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ For a first real decision, follow the [complete CPU walkthrough](../docs/getting
Preview checkpoint and Qwen base, start the text-or-image `choice` worker and verify the frontend.
- [Cua-S1 4B 0.2 native text worker](cua_s1/native.md): build the CUDA library and
the Rust worker, export the merged weights and start the worker.
- [Cua-S1 native screenshot worker](cua_s1/native_screenshot.md): export the
multimodal language checkpoint and serve one-image decisions with Rust/CUDA.
- [Open-Jev native text worker](open_jev/native.md): export the merged text
backbone and trained decision head of Open-Jev-27B-v1.1 or Open-Jev-9B, then
serve with Rust and CUDA. [Open-Jev-9B validation](open_jev/validation-9b.md)
Expand Down
4 changes: 2 additions & 2 deletions recipe/cua_s1/native.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ For the local CUDA Graph experiment, also set `CUA_S1_GRAPH=1`. The first use of
each exact ordered prompt-length vector warms the GEMM plans and captures the forward pass;
later requests replay it with freshly uploaded token ids. Each language mode keeps a separate 64-entry FIFO cache. Growing the scratch allocation clears the captures before freeing
their buffers. Capture adds first-use latency; leave the variable unset to use
the eager control. Rebuild both the worker and CUDA library together (ABI 6).
the eager control. Rebuild both the worker and CUDA library together (ABI 8).
If capture fails, the worker returns the completed eager result and disables
Graph capture/replay for its remaining lifetime, logging the failure to stderr.

Expand All @@ -42,5 +42,5 @@ CUA_S1_CUDA_LIB=$PWD/target/release/libqwen3_5_cuda.so \
```

For the screenshot worker, [the vision Graph recipe](native_vision_graph.md)
documents `CUA_S1_VISION_GRAPH`, retained single-grid scratch, memory/lifetime
documents `CUA_S1_VISION_GRAPH`, retained four-grid FIFO scratch, memory/lifetime
limits and the real CUDA capture-failure wrapper build and test commands.
96 changes: 96 additions & 0 deletions recipe/cua_s1/native_screenshot.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# Native Cua-S1 screenshot serving

The `omni-cua-s1-vision` worker serves the Cua-S1 4B 0.2 `multimodal`
adapter with native Rust/CUDA vision and language execution. It accepts one
inline PNG/JPEG screenshot and 1–8 independent `choice` questions. The vision
encoder runs once per request; each question keeps its own language prompt.
See the [model contract](../../src/models/cua_s1/README.md) for pinned revisions
and the [vision Graph recipe](native_vision_graph.md) for device tests and controls.

## Prepare and build

Prepare the pinned base and adapter using the [reference setup](text.md), keeping
the upstream `weights.lock.json` alongside the downloaded artifacts. Use the
reference Python environment only for this one-time language export. The vision
worker uses the original BF16 vision tensors and unmerged FP32 vision LoRA:

```sh
PYTHONPATH=src .venv/bin/python recipe/cua_s1/export_multimodal_language.py \
--base weights/Qwen3.5-4B \
--adapter weights/cua-s1-4b-0.2/multimodal \
--out weights/cua-s1-multimodal-language

src/backends/cuda/qwen3_5/build.sh target/release 89
cargo build --release --locked -p omni-cua-s1-native -p omni-jev
```

The export refuses an existing output directory. Keep the export and original
weights immutable while serving. `89` targets the validated RTX 4090; choose your
device's supported compute capability when building. Rebuild the CUDA library
and every native consumer together when the ABI changes. This branch requires
ABI 8, including the default host-feature path.

## Start the worker and frontend

```sh
CUA_S1_BASE=weights/Qwen3.5-4B \
CUA_S1_VISION_ADAPTER=weights/cua-s1-4b-0.2/multimodal \
CUA_S1_MODEL=weights/cua-s1-multimodal-language \
CUA_S1_CUDA_LIB=$PWD/target/release/libqwen3_5_cuda.so \
CUA_S1_HOST=127.0.0.1 CUA_S1_PORT=8000 \
CUA_S1_GRAPH=0 CUA_S1_VISION_GRAPH=0 CUA_S1_DEVICE_FEATURES=0 \
target/release/omni-cua-s1-vision
```

In another terminal, start the existing Rust frontend:

```sh
OMNI_JEV_BIND=127.0.0.1:8080 \
OMNI_JEV_BACKEND_URL=http://127.0.0.1:8000 target/release/omni-jev
curl --fail http://127.0.0.1:8080/health
```

Create a request from a local screenshot, then send it through the frontend:

```sh
python3 - screenshot.png > /tmp/cua-screenshot-request.json <<'PY'
import base64, json, pathlib, sys
image = base64.b64encode(pathlib.Path(sys.argv[1]).read_bytes()).decode()
json.dump({
"model": "cua-s1-4b-0.2",
"state": {"image": "data:image/png;base64," + image},
"questions": {"pick": {
"type": "choice", "instructions": "Keep the files.",
"criteria": {"delete": "Click Delete", "cancel": "Click Cancel"}
}}
}, sys.stdout)
PY
curl --fail-with-body http://127.0.0.1:8080/v1/systemone \
-H 'Content-Type: application/json' \
--data-binary @/tmp/cua-screenshot-request.json
```

Use matching `data:image/jpeg;base64,` for a JPEG. Returned option probabilities
and the selected key depend on the screenshot. The response identifies the
multimodal adapter and reports prompt-token usage with zero generated tokens.

## Execution controls and limits

`CUA_S1_GRAPH=1` enables host-feature language Graph replay;
`CUA_S1_VISION_GRAPH=1` independently enables vision Graph replay.
`CUA_S1_DEVICE_FEATURES=1` keeps vision features on the GPU and copies image rows
directly into language scratch. Its language path remains eager; it can be paired
with either vision Graph setting. All three controls default off. See the
[resident-feature contract](native_vision_graph.md#gpu-resident-vision-features)
for strict control parsing, synchronization and reproduction commands.

Requests are serialized on the loaded GPU executor. HTTP concurrency includes
preparation and queueing; there is no cross-request GPU batch. Input is limited
to an 8 MiB body, 4 MiB image file, dimensions up to 2048 on each axis
and 1,048,576 pixels, one image, and 1–26 options per question. Animated images,
remote image URLs, multi-image state, `score`, `noul` and native Metal are unsupported.

Recorded RTX 4090 evidence covers native screenshot parity and Graph regressions;
the resident-feature comparison covers host/device output equality on its fixed
real-screenshot corpus. These scopes do not establish general GUI task accuracy
or native equivalence to every reference precision and environment.
94 changes: 91 additions & 3 deletions recipe/cua_s1/native_vision_graph.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,9 @@ Use the pinned base and multimodal adapter revisions listed in the
[model contract](../../src/models/cua_s1/README.md#pinned-revisions), and the
verified language export produced by `export_multimodal_language.py`. Keep all
checkpoint files immutable during execution. Build the worker and the current
CUDA library together; this change adds no CUDA ABI symbols.
CUDA library together for **ABI 8**. Resident feature validation adds
`cs1_get_device` and `cs1_bf16_finite`; older libraries are rejected even when
resident features are disabled.

```sh
src/backends/cuda/qwen3_5/build.sh target/release 89
Expand All @@ -49,13 +51,99 @@ CUA_S1_VISION_GRAPH=1 target/release/omni-cua-s1-vision
Set `CUA_S1_PORT` to choose the listening port (default 8000). Only the existing
single-image native CUDA path is covered. Python workers and Metal are unchanged.

## GPU-resident vision features

Set `CUA_S1_DEVICE_FEATURES=1` before loading `omni-cua-s1-vision` to keep the
vision merger's BF16 output on the GPU. Unset or `0` selects the host path;
every other explicit value, including an empty or non-Unicode value, fails
before checkpoint or CUDA loading. Export the control before running the launch
command above:

```sh
export CUA_S1_DEVICE_FEATURES=1
```

This switch is independent of `CUA_S1_VISION_GRAPH`. Vision can run eagerly or
replay a Graph; the resident-feature language path remains eager in both modes.
The host-feature language path retains the independent `CUA_S1_GRAPH` control.
The resident path reuses one vision output across all questions in the request
and copies contiguous image-placeholder rows directly into language scratch.
It avoids the full feature download and per-question feature upload. It still
downloads the final hidden state for the existing CPU letter readout.

`VisionModel::forward_device` returns a borrowed, row-major BF16 matrix after
finishing the vision stream. The borrow prevents producer scratch mutation or
retirement while the features are in use. Language execution validates the
exact allocation extent, shape, device, placeholder order and positions, then
drains its stream on success or error before the borrow can end. Each question
checks finite BF16 values with a GPU scan and a four-byte result download;
flag allocation and synchronization overhead remain. The feature remains
disabled by default; reduced transfers alone do not establish a general
latency or throughput improvement.

`DeviceBf16Matrix::download` and `VisionEngine::score_device` provide diagnostic
feature/readout comparisons. Serving uses the resident rows without downloading
the complete matrix. The root tests below exercise both diagnostic APIs.

### Resident-feature regressions

Weight-free CPU checks run in ordinary CI:

```sh
cargo test --locked -p omni-cua-s1-native --test device_controls
cargo test --locked -p omni-qwen3-5-native --test device_features
cargo test --locked -p omni-qwen3-5-native --lib device_input_tests
```

GPU tests compile in CI but are ignored by default. On a reserved GPU, export
the four checkpoint/library variables from the launch command and run serially.
The finite-scan test needs only `CUA_S1_CUDA_LIB`; the other tests need all four:

```sh
cargo test --release --locked -p omni-qwen3-5-native --lib device_tests \
-- --ignored --test-threads=1
CUA_S1_VISION_GRAPH=0 cargo test --release --locked -p omni-cua-s1-native \
--lib device_feature_tests -- --ignored --test-threads=1
CUA_S1_VISION_GRAPH=1 cargo test --release --locked -p omni-cua-s1-native \
--lib device_feature_tests -- --ignored --test-threads=1
cargo test --release --locked -p omni-qwen3-5-native --lib device_vision_graph_ \
-- --ignored --test-threads=1
```

These tests compare feature bits, hidden states, logits, probabilities and
complete responses across changed pixels/goals, repeated questions, different
grids with equal patch counts, scratch growth and short-request recovery.
They also reject invalid metadata and non-finite features. For capture failure,
build the wrapper in the next section and run in a separate process:

```sh
CUA_S1_CUDA_LIB=/tmp/cua-fail-capture.so CUA_S1_TEST_CAPTURE_FAILURE=1 \
cargo test --release --locked -p omni-qwen3-5-native \
--lib device_vision_capture_failure_ -- --ignored --test-threads=1
```

The queued-copy wrapper enqueues one real D2D copy, returns an injected error,
and records a successful real stream drain. The test checks that record before
releasing the vision view, then verifies request recovery across grid changes.
Use an absolute real-library path and a new evidence path for each run:

```sh
export CUA_S1_REAL_CUDA_LIB="$CUA_S1_CUDA_LIB"
g++ -std=c++17 -shared -fPIC -Wall -Wextra tests/cua_s1/fail_device_copy.cpp \
-Wl,--no-as-needed "$CUA_S1_REAL_CUDA_LIB" -ldl -o /tmp/cua-fail-device-copy.so
CUA_S1_CUDA_LIB=/tmp/cua-fail-device-copy.so CUA_S1_TEST_COPY_FAILURE=1 \
CUA_S1_TEST_COPY_FAILURE_EVIDENCE=/tmp/fresh-copy-drain.txt \
cargo test --release --locked -p omni-cua-s1-native \
--lib device_copy_failure_tests -- --ignored --test-threads=1 --nocapture
```

## Regressions and native measurements

Export the same `CUA_S1_BASE`, `CUA_S1_VISION_ADAPTER`, `CUA_S1_MODEL`, and
`CUA_S1_CUDA_LIB` variables for these commands:

```sh
cargo test --release --locked -p omni-qwen3-5-native --lib vision_graph_ \
cargo test --release --locked -p omni-qwen3-5-native --lib vision::model::vision_tests::vision_graph_ \
-- --ignored --test-threads=1 --nocapture
CUA_S1_GRAPH=0 CUA_S1_VISION_GRAPH=0 cargo run --release --locked -p omni-cua-s1-native \
--example vision_graph_bench -- /tmp/vision-eager.json
Expand All @@ -75,7 +163,7 @@ g++ -std=c++17 -shared -fPIC -Wall -Wextra tests/cua_s1/fail_graph_capture.cpp \
-o /tmp/cua-fail-capture.so
CUA_S1_CUDA_LIB=/tmp/cua-fail-capture.so CUA_S1_TEST_CAPTURE_FAILURE=1 \
CUA_S1_GRAPH_TRACE=0 cargo test --release --locked -p omni-qwen3-5-native \
--lib vision_capture_failure_ -- --ignored --test-threads=1 --nocapture
--lib vision::model::vision_tests::vision_capture_failure_ -- --ignored --test-threads=1 --nocapture
```

The root GPU tests cover changed pixels, exact-grid residency and eviction, retirement,
Expand Down
Loading
Loading