Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
b1e216f
Merge #98 and #99 (fixed-algorithm GEMMs, run_window) for the worker
twu3202 Oct 10, 2026
923cb16
Merge #117 (shared vision) for the worker
twu3202 Oct 10, 2026
fc27b75
omnijev: add the native eager worker
twu3202 Oct 10, 2026
5cf7c4e
omnijev: hold the admission slot until the request's work ends
twu3202 Oct 10, 2026
ee1bf55
omnijev: reuse the prefix within a request
twu3202 Oct 10, 2026
b5fdc74
omnijev: say the prefix states keep only their buffers across requests
twu3202 Oct 10, 2026
d96f843
Merge #99 and main into omnijev-worker
twu3202 Oct 11, 2026
a8f523a
omnijev: list the native worker in the qwen3_5 backend manifest
twu3202 Oct 11, 2026
be68c68
Merge omnijev-worker into omnijev-reuse
twu3202 Oct 11, 2026
dac6ad6
Merge #99 and main into omnijev-worker
twu3202 Oct 13, 2026
8cf558c
Merge omnijev-worker into omnijev-reuse
twu3202 Oct 13, 2026
27bec29
Merge #117 into omnijev-worker
twu3202 Oct 14, 2026
b220f47
Merge main into omnijev-worker
twu3202 Oct 14, 2026
c2a5759
Merge omnijev-worker into omnijev-reuse
twu3202 Oct 14, 2026
b81a42b
Merge prefix-shared-executor into omnijev-worker
twu3202 Oct 21, 2026
1e92d82
Merge omnijev-worker into omnijev-reuse
twu3202 Oct 21, 2026
a58fb24
Merge prefix-shared-executor into omnijev-worker
twu3202 Oct 21, 2026
d7418ce
Refuse other execution profiles in forward_multimodal_readout
twu3202 Oct 21, 2026
8b8882e
Merge omnijev-worker into omnijev-reuse
twu3202 Oct 21, 2026
b270a56
Merge prefix-shared-executor into omnijev-worker
twu3202 Oct 22, 2026
3d73cda
Merge omnijev-worker into omnijev-reuse
twu3202 Oct 22, 2026
a4a9d2d
Merge main into omnijev-worker
twu3202 Oct 25, 2026
6c60fb8
Merge omnijev-worker into omnijev-reuse
twu3202 Oct 25, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ runs as a Python worker or as a native worker on CUDA. Open-Jev-27B-v1.1
and Open-Jev-9B run on the same native Rust/CUDA worker. Cua-S1 also has a
Python screenshot worker, Valen-Preview-0923 has a Python reference worker for
text and one-image state, CLM has a stub-encoder contract recipe, and OmniJev-4B
v1.1 has its checkpoint export and CPU contract, with no worker yet. D3
v1.1 runs on a native Rust/CUDA worker. D3
(Decision 3.0) has a Python reference worker that runs the checkpoint's own
pinned runtime for text, image and video state:

Expand All @@ -179,7 +179,7 @@ pinned runtime for text, image and video state:
| autotrust/JEV-27B-VL | [Experimental Rust/CUDA worker](recipe/jev_vl/README.md); single-question text, online inline PNG/JPEG and offline-preencoded image inputs; [validation scope and limits](recipe/jev_vl/validation.md) |
| CLM-v0.1-8B | [External worker with a CPU stub encoder](recipe/clm/README.md); contract checks only, real Qwen3-8B decisions unverified by this recipe |
| LFM2.5-350M | [Transformers worker](recipe/lfm2/README.md); text state and independent choice questions |
| OmniJev-4B v1.1 | [Checkpoint export and CPU contract](recipe/omnijev/README.md) only; the native worker is in progress ([#114](https://github.com/ThinkFlowLab/system1-omni/issues/114)) |
| OmniJev-4B v1.1 | [Native Rust/CUDA worker](recipe/omnijev/native.md), eager: one image, Choice, Noul and Score; [reference comparison on sm_89](recipe/omnijev/native.md#comparison-with-the-reference) |
| D3 (Decision 3.0) | [Python reference worker](recipe/d3/reference.md) on the checkpoint's pinned runtime; `choice`/`noul`/`score` over text, image and video state; author-run on A100 for text and media serving (see the integration and media PRs) |

[Supported models and hardware](docs/supported-models.md) lists the devices
Expand Down
2 changes: 1 addition & 1 deletion docs/supported-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Models that are being added are also tracked in issues labeled [new model](https
| CLM-v0.1-8B | [External `clm-serve` recipe](../recipe/clm/README.md) with a CPU stub embeddings server | **Stub-encoder contract checks only** ([#23](https://github.com/ThinkFlowLab/system1-omni/pull/23)); not real Qwen3-8B decisions | Real encoder unverified by the merged recipe | Unverified | Python, upstream CLM and head checkpoint; a real encoder requires a separate embeddings server |
| LFM2.5-350M | [Transformers choice worker](../recipe/lfm2/README.md) | Tiny-model regression tests; full checkpoint unverified | L40S FP16 [recorded execution checks](../recipe/lfm2/RESULTS.md) | Unverified | Python 3.12, [pinned dependencies](../src/models/lfm2/requirements.lock.txt), explicit candidate batch size |
| CLM-v0.1-8B | [Native Rust engine](../src/models/clm/README.md), `clm-run`, in front of a Qwen3-8B `/v1/embeddings` server | Engine-only agreement **4.5e-06** on identical vectors ([#29](https://github.com/ThinkFlowLab/system1-omni/pull/29), [validation](../recipe/clm/native/VALIDATION.md)); contract checks documented ([#23](https://github.com/ThinkFlowLab/system1-omni/pull/23)) | **Validated on compute capability 8.9** ([#29](https://github.com/ThinkFlowLab/system1-omni/pull/29), [validation](../recipe/clm/native/VALIDATION.md)) | Unverified | Rust to build and a separate Qwen3-8B `/v1/embeddings` server for the encoder; the engine itself has no device-specific code |
| OmniJev-4B v1.1 | None yet: [checkpoint export and CPU contract](../recipe/omnijev/README.md); the native worker is in progress ([#114](https://github.com/ThinkFlowLab/system1-omni/issues/114)) | Request contract, preparation, heads and finishing checked against the reference ([#120](https://github.com/ThinkFlowLab/system1-omni/pull/120)); no inference | Not yet | Not supported | Python 3.10 reference environment for the export and fixtures |
| OmniJev-4B v1.1 | [Native Rust/CUDA worker](../recipe/omnijev/native.md) on the shared Qwen3.5 vision and language path, eager | Contract, preparation, pixels, heads and finishing checked against the reference; no CPU inference | Compared with the reference on compute capability 8.9 for [140 answers](../recipe/omnijev/native.md#comparison-with-the-reference) | Not supported | Compute capability 8.0 or newer, CUDA toolkit to build, the export from `export.py` |
| D3 (Decision 3.0) | [Python reference worker](../recipe/d3/reference.md) on the checkpoint's pinned runtime — text, image and video state over `choice`/`noul`/`score`, with the deliberate transport differences documented in the recipe; the native worker is in progress ([#136](https://github.com/ThinkFlowLab/system1-omni/issues/136)) | Unverified | Author-run on NVIDIA A100 (sm_80): serving smoke and reference parity for text and media (exact decisions, max delta 0.0); evidence in the integration and media PRs. Owner: @MrDongsls. | Unverified | Python 3.12, the [pinned runtime requirements](../recipe/d3/requirements.txt), the [pinned package](https://huggingface.co/vllm-sr/d3) (~52 GB); [media budgets and the 64 MiB total request cap](../recipe/d3/reference.md) |

- **Validated:** covered by the recipe on `main` or by the checks in the linked merged pull request.
Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ nav:
- JEV-27B-VL experimental worker: recipe/jev_vl/README.md
- JEV-27B-VL validation status: recipe/jev_vl/validation.md
- OmniJev preparation: recipe/omnijev/README.md
- OmniJev native worker: recipe/omnijev/native.md
- CLM stub-encoder contract: recipe/clm/README.md
- LFM2.5-350M choice worker: recipe/lfm2/README.md
- D3 reference worker: recipe/d3/reference.md
Expand Down
3 changes: 2 additions & 1 deletion recipe/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ For a first real decision, follow the [complete CPU walkthrough](../docs/getting
- [JEV-27B-VL experimental worker](jev_vl/README.md): native text and preencoded
image decisions; bounded H800 validation with explicit deployment limits.
- [OmniJev-4B v1.1 preparation](omnijev/README.md): export the checkpoint and
regenerate the CPU fixtures; the native worker is not available yet.
regenerate the CPU fixtures; [native worker](omnijev/native.md): serve it on CUDA
and compare it with the reference.
- [CLM behind the frontend](clm/README.md): run CLM's own server behind the frontend on
CPU with a stub encoder, and what the response comparison has to allow for.
- [D3 reference worker](d3/reference.md): download the pinned vllm-sr/d3 package,
Expand Down
12 changes: 6 additions & 6 deletions recipe/omnijev/README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# OmniJev-4B v1.1 preparation

This recipe prepares OmniJev-4B v1.1 ([#114](https://github.com/ThinkFlowLab/system1-omni/issues/114))
for the native worker: it exports the checkpoint and regenerates the CPU fixtures
that the contract and preparation tests use. The native worker itself is not
available yet; [the model README](../../src/models/omnijev/README.md) describes what
is implemented. Run the commands from the repository root.
for [the native worker](native.md): it exports the checkpoint and regenerates the CPU
fixtures that the contract, preparation and pixel tests use. [The model
README](../../src/models/omnijev/README.md) describes what is implemented. Run the
commands from the repository root.

## Reference environment

Expand Down Expand Up @@ -65,8 +65,8 @@ OMNIJEV_CHECKPOINT=$PWD/omnijev-v1.1 OMNIJEV_EXPORT=$PWD/omnijev-export \
cargo test --locked -p omni-omnijev-native -- --include-ignored
```

The generator runs the reference's preparation, layout, rotary positions, heads and
finishing without loading the backbone. The ordinary tests need only the committed
The generator runs the reference's image loading and processor, preparation, layout,
rotary positions, heads and finishing without loading the backbone. The ordinary tests need only the committed
fixtures; the opt-in ones compare token ids with the checkpoint's tokenizer and the
heads with the export. Cargo runs the tests from the crate's directory, so the two
paths must be absolute.
242 changes: 242 additions & 0 deletions recipe/omnijev/gate_questions.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,242 @@
{
"one_token": {
"type": "choice",
"instructions": "Which button should be clicked next?",
"options": [
{
"key": "click OK",
"text": "click OK"
},
{
"key": "click Cancel",
"text": "click Cancel"
},
{
"key": "click Close",
"text": "click Close"
},
{
"key": "click Help",
"text": "click Help"
},
{
"abstain": true
}
]
},
"long_start": {
"type": "choice",
"instructions": "What should the user do next?",
"options": [
{
"key": "first",
"text": "open the settings menu at the top of the screen and then select the first item"
},
{
"key": "second",
"text": "open the settings menu at the top of the screen and then select the second item"
},
{
"key": "third",
"text": "open the settings menu at the top of the screen and then select the third item"
},
{
"abstain": true
}
]
},
"no_fit": {
"type": "choice",
"instructions": "Which animal is shown in the picture?",
"options": [
{
"key": "a giraffe",
"text": "a giraffe"
},
{
"key": "a whale",
"text": "a whale"
},
{
"key": "a penguin",
"text": "a penguin"
},
{
"abstain": true
}
]
},
"score10": {
"type": "score",
"instructions": "How bright is the image overall?",
"levels": [
"pitch black",
"very dark",
"dark",
"dim",
"slightly dim",
"neutral",
"slightly bright",
"bright",
"very bright",
"blinding"
]
},
"choice32": {
"type": "choice",
"instructions": "Which word best describes the image?",
"options": [
{
"key": "game",
"text": "game"
},
{
"key": "chart",
"text": "chart"
},
{
"key": "map",
"text": "map"
},
{
"key": "menu",
"text": "menu"
},
{
"key": "photo",
"text": "photo"
},
{
"key": "diagram",
"text": "diagram"
},
{
"key": "table",
"text": "table"
},
{
"key": "form",
"text": "form"
},
{
"key": "chat",
"text": "chat"
},
{
"key": "email",
"text": "email"
},
{
"key": "video",
"text": "video"
},
{
"key": "code",
"text": "code"
},
{
"key": "terminal",
"text": "terminal"
},
{
"key": "calendar",
"text": "calendar"
},
{
"key": "document",
"text": "document"
},
{
"key": "spreadsheet",
"text": "spreadsheet"
},
{
"key": "slide",
"text": "slide"
},
{
"key": "browser",
"text": "browser"
},
{
"key": "robot",
"text": "robot"
},
{
"key": "board",
"text": "board"
},
{
"key": "phone",
"text": "phone"
},
{
"key": "street",
"text": "street"
},
{
"key": "kitchen",
"text": "kitchen"
},
{
"key": "office",
"text": "office"
},
{
"key": "forest",
"text": "forest"
},
{
"key": "beach",
"text": "beach"
},
{
"key": "city",
"text": "city"
},
{
"key": "animal",
"text": "animal"
},
{
"key": "person",
"text": "person"
},
{
"key": "car",
"text": "car"
},
{
"key": "screen",
"text": "screen"
},
{
"key": "error",
"text": "error"
},
{
"abstain": true
}
]
},
"noul_day": {
"type": "noul",
"instructions": "The image was taken during the day."
},
"noul_people": {
"type": "noul",
"instructions": "There are more than three people in the image."
},
"noul_region": {
"type": "noul",
"instructions": "There is text in this region.",
"region": {
"box": [
100,
100,
700,
400
]
}
}
}
Loading
Loading