Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ runs as a Python worker or as a native worker on CUDA. Open-Jev-27B-v1.1
and Open-Jev-9B run on the same native Rust/CUDA worker. Cua-S1 also has a
Python screenshot worker, Valen-Preview-0923 has a Python reference worker for
text and one-image state, CLM has a stub-encoder contract recipe, and OmniJev-4B
v1.1 has its checkpoint export and CPU contract, with no worker yet. D3
v1.1 runs on a native Rust/CUDA worker. D3
(Decision 3.0) has a Python reference worker that runs the checkpoint's own
pinned runtime for text, image and video state:

Expand All @@ -179,7 +179,7 @@ pinned runtime for text, image and video state:
| autotrust/JEV-27B-VL | [Experimental Rust/CUDA worker](recipe/jev_vl/README.md); single-question text, online inline PNG/JPEG and offline-preencoded image inputs; [validation scope and limits](recipe/jev_vl/validation.md) |
| CLM-v0.1-8B | [External worker with a CPU stub encoder](recipe/clm/README.md); contract checks only, real Qwen3-8B decisions unverified by this recipe |
| LFM2.5-350M | [Transformers worker](recipe/lfm2/README.md); text state and independent choice questions |
| OmniJev-4B v1.1 | [Checkpoint export and CPU contract](recipe/omnijev/README.md) only; the native worker is in progress ([#114](https://github.com/ThinkFlowLab/system1-omni/issues/114)) |
| OmniJev-4B v1.1 | [Native Rust/CUDA worker](recipe/omnijev/native.md), eager: one image, Choice, Noul and Score; [reference comparison on sm_89](recipe/omnijev/native.md#comparison-with-the-reference) |
| D3 (Decision 3.0) | [Python reference worker](recipe/d3/reference.md) on the checkpoint's pinned runtime; `choice`/`noul`/`score` over text, image and video state; author-run on A100 for text and media serving (see the integration and media PRs) |

[Supported models and hardware](docs/supported-models.md) lists the devices
Expand Down
2 changes: 1 addition & 1 deletion docs/supported-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Models that are being added are also tracked in issues labeled [new model](https
| CLM-v0.1-8B | [External `clm-serve` recipe](../recipe/clm/README.md) with a CPU stub embeddings server | **Stub-encoder contract checks only** ([#23](https://github.com/ThinkFlowLab/system1-omni/pull/23)); not real Qwen3-8B decisions | Real encoder unverified by the merged recipe | Unverified | Python, upstream CLM and head checkpoint; a real encoder requires a separate embeddings server |
| LFM2.5-350M | [Transformers choice worker](../recipe/lfm2/README.md) | Tiny-model regression tests; full checkpoint unverified | L40S FP16 [recorded execution checks](../recipe/lfm2/RESULTS.md) | Unverified | Python 3.12, [pinned dependencies](../src/models/lfm2/requirements.lock.txt), explicit candidate batch size |
| CLM-v0.1-8B | [Native Rust engine](../src/models/clm/README.md), `clm-run`, in front of a Qwen3-8B `/v1/embeddings` server | Engine-only agreement **4.5e-06** on identical vectors ([#29](https://github.com/ThinkFlowLab/system1-omni/pull/29), [validation](../recipe/clm/native/VALIDATION.md)); contract checks documented ([#23](https://github.com/ThinkFlowLab/system1-omni/pull/23)) | **Validated on compute capability 8.9** ([#29](https://github.com/ThinkFlowLab/system1-omni/pull/29), [validation](../recipe/clm/native/VALIDATION.md)) | Unverified | Rust to build and a separate Qwen3-8B `/v1/embeddings` server for the encoder; the engine itself has no device-specific code |
| OmniJev-4B v1.1 | None yet: [checkpoint export and CPU contract](../recipe/omnijev/README.md); the native worker is in progress ([#114](https://github.com/ThinkFlowLab/system1-omni/issues/114)) | Request contract, preparation, heads and finishing checked against the reference ([#120](https://github.com/ThinkFlowLab/system1-omni/pull/120)); no inference | Not yet | Not supported | Python 3.10 reference environment for the export and fixtures |
| OmniJev-4B v1.1 | [Native Rust/CUDA worker](../recipe/omnijev/native.md) on the shared Qwen3.5 vision and language path, eager | Contract, preparation, pixels, heads and finishing checked against the reference; no CPU inference | Compared with the reference on compute capability 8.9 for [140 answers](../recipe/omnijev/native.md#comparison-with-the-reference) | Not supported | Compute capability 8.0 or newer, CUDA toolkit to build, the export from `export.py` |
| D3 (Decision 3.0) | [Python reference worker](../recipe/d3/reference.md) on the checkpoint's pinned runtime — text, image and video state over `choice`/`noul`/`score`, with the deliberate transport differences documented in the recipe; the native worker is in progress ([#136](https://github.com/ThinkFlowLab/system1-omni/issues/136)) | Unverified | Author-run on NVIDIA A100 (sm_80): serving smoke and reference parity for text and media (exact decisions, max delta 0.0); evidence in the integration and media PRs. Owner: @MrDongsls. | Unverified | Python 3.12, the [pinned runtime requirements](../recipe/d3/requirements.txt), the [pinned package](https://huggingface.co/vllm-sr/d3) (~52 GB); [media budgets and the 64 MiB total request cap](../recipe/d3/reference.md) |

- **Validated:** covered by the recipe on `main` or by the checks in the linked merged pull request.
Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ nav:
- JEV-27B-VL experimental worker: recipe/jev_vl/README.md
- JEV-27B-VL validation status: recipe/jev_vl/validation.md
- OmniJev preparation: recipe/omnijev/README.md
- OmniJev native worker: recipe/omnijev/native.md
- CLM stub-encoder contract: recipe/clm/README.md
- LFM2.5-350M choice worker: recipe/lfm2/README.md
- D3 reference worker: recipe/d3/reference.md
Expand Down
3 changes: 2 additions & 1 deletion recipe/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ For a first real decision, follow the [complete CPU walkthrough](../docs/getting
- [JEV-27B-VL experimental worker](jev_vl/README.md): native text and preencoded
image decisions; bounded H800 validation with explicit deployment limits.
- [OmniJev-4B v1.1 preparation](omnijev/README.md): export the checkpoint and
regenerate the CPU fixtures; the native worker is not available yet.
regenerate the CPU fixtures; [native worker](omnijev/native.md): serve it on CUDA
and compare it with the reference.
- [CLM behind the frontend](clm/README.md): run CLM's own server behind the frontend on
CPU with a stub encoder, and what the response comparison has to allow for.
- [D3 reference worker](d3/reference.md): download the pinned vllm-sr/d3 package,
Expand Down
12 changes: 6 additions & 6 deletions recipe/omnijev/README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# OmniJev-4B v1.1 preparation

This recipe prepares OmniJev-4B v1.1 ([#114](https://github.com/ThinkFlowLab/system1-omni/issues/114))
for the native worker: it exports the checkpoint and regenerates the CPU fixtures
that the contract and preparation tests use. The native worker itself is not
available yet; [the model README](../../src/models/omnijev/README.md) describes what
is implemented. Run the commands from the repository root.
for [the native worker](native.md): it exports the checkpoint and regenerates the CPU
fixtures that the contract, preparation and pixel tests use. [The model
README](../../src/models/omnijev/README.md) describes what is implemented. Run the
commands from the repository root.

## Reference environment

Expand Down Expand Up @@ -65,8 +65,8 @@ OMNIJEV_CHECKPOINT=$PWD/omnijev-v1.1 OMNIJEV_EXPORT=$PWD/omnijev-export \
cargo test --locked -p omni-omnijev-native -- --include-ignored
```

The generator runs the reference's preparation, layout, rotary positions, heads and
finishing without loading the backbone. The ordinary tests need only the committed
The generator runs the reference's image loading and processor, preparation, layout,
rotary positions, heads and finishing without loading the backbone. The ordinary tests need only the committed
fixtures; the opt-in ones compare token ids with the checkpoint's tokenizer and the
heads with the export. Cargo runs the tests from the crate's directory, so the two
paths must be absolute.
242 changes: 242 additions & 0 deletions recipe/omnijev/gate_questions.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,242 @@
{
"one_token": {
"type": "choice",
"instructions": "Which button should be clicked next?",
"options": [
{
"key": "click OK",
"text": "click OK"
},
{
"key": "click Cancel",
"text": "click Cancel"
},
{
"key": "click Close",
"text": "click Close"
},
{
"key": "click Help",
"text": "click Help"
},
{
"abstain": true
}
]
},
"long_start": {
"type": "choice",
"instructions": "What should the user do next?",
"options": [
{
"key": "first",
"text": "open the settings menu at the top of the screen and then select the first item"
},
{
"key": "second",
"text": "open the settings menu at the top of the screen and then select the second item"
},
{
"key": "third",
"text": "open the settings menu at the top of the screen and then select the third item"
},
{
"abstain": true
}
]
},
"no_fit": {
"type": "choice",
"instructions": "Which animal is shown in the picture?",
"options": [
{
"key": "a giraffe",
"text": "a giraffe"
},
{
"key": "a whale",
"text": "a whale"
},
{
"key": "a penguin",
"text": "a penguin"
},
{
"abstain": true
}
]
},
"score10": {
"type": "score",
"instructions": "How bright is the image overall?",
"levels": [
"pitch black",
"very dark",
"dark",
"dim",
"slightly dim",
"neutral",
"slightly bright",
"bright",
"very bright",
"blinding"
]
},
"choice32": {
"type": "choice",
"instructions": "Which word best describes the image?",
"options": [
{
"key": "game",
"text": "game"
},
{
"key": "chart",
"text": "chart"
},
{
"key": "map",
"text": "map"
},
{
"key": "menu",
"text": "menu"
},
{
"key": "photo",
"text": "photo"
},
{
"key": "diagram",
"text": "diagram"
},
{
"key": "table",
"text": "table"
},
{
"key": "form",
"text": "form"
},
{
"key": "chat",
"text": "chat"
},
{
"key": "email",
"text": "email"
},
{
"key": "video",
"text": "video"
},
{
"key": "code",
"text": "code"
},
{
"key": "terminal",
"text": "terminal"
},
{
"key": "calendar",
"text": "calendar"
},
{
"key": "document",
"text": "document"
},
{
"key": "spreadsheet",
"text": "spreadsheet"
},
{
"key": "slide",
"text": "slide"
},
{
"key": "browser",
"text": "browser"
},
{
"key": "robot",
"text": "robot"
},
{
"key": "board",
"text": "board"
},
{
"key": "phone",
"text": "phone"
},
{
"key": "street",
"text": "street"
},
{
"key": "kitchen",
"text": "kitchen"
},
{
"key": "office",
"text": "office"
},
{
"key": "forest",
"text": "forest"
},
{
"key": "beach",
"text": "beach"
},
{
"key": "city",
"text": "city"
},
{
"key": "animal",
"text": "animal"
},
{
"key": "person",
"text": "person"
},
{
"key": "car",
"text": "car"
},
{
"key": "screen",
"text": "screen"
},
{
"key": "error",
"text": "error"
},
{
"abstain": true
}
]
},
"noul_day": {
"type": "noul",
"instructions": "The image was taken during the day."
},
"noul_people": {
"type": "noul",
"instructions": "There are more than three people in the image."
},
"noul_region": {
"type": "noul",
"instructions": "There is text in this region.",
"region": {
"box": [
100,
100,
700,
400
]
}
}
}
Loading
Loading