Fast decision-model serving for agents.
How It Works · First Decision · Demo · Supported Models · 中文 · Documentation · Benchmarks · Roadmap · Contributing
System1-Omni serves decision models for agents: route a support ticket to billing, score its urgency, or decide whether it asks for a refund. Models return structured decisions from a prefill pass. Start with the complete CPU walkthrough, inspect the real request and response, and check the model and hardware matrix.
The community-maintained engine combines a Rust frontend with model-owned execution and CUDA backends. The target architecture separates processing and scheduling from model execution; a native Metal backend is planned, while LAYA already has a Python worker for Apple GPUs through PyTorch MPS.
The Rust frontend forwards requests to a separately running model worker. The
Cua-S1 4B 0.2 text adapter, Open-Jev-27B-v1.1 and Open-Jev-9B have native
workers using shared CUDA kernels in this repository.
The LFM2.5-350M Transformers worker owns candidate scoring and hybrid cache branching.
- 2026-10-03: Added Open-Jev-27B-v1.1
support through a native Rust/CUDA worker: 7.47× faster than raw HF Transformers
by mean warm HTTP latency, 362.21→48.50 ms on one H200. Measured over
74 single-candidate JevBench
noulrequests per pass, with two measured passes per backend (BF16, concurrency 1). See the HF Transformers baseline, results and OpenJev-Fast comparison.
- Rust serving frontend. API, request lifecycle, and response delivery through a small engine interface, forwarding requests to separately running model workers.
- Model-owned execution. Model executors own weights, forward passes, learned heads, device state, and kernel selection. Native workers have separate processing and executor modules, with shared FIFO admission and blocking dispatch in the native runtime.
- Native CUDA workers. The Cua-S1 4B 0.2
textadapter, Open-Jev-27B-v1.1, Open-Jev-9B, and StartLux-Decision v1.1 4B Instant run as native workers with shared CUDA kernels. Cua-S1 also has a Python worker that serves as the correctness reference. The StartLux worker is text-only Instant; its weights are CC BY-NC 4.0 and its GPU parity is unverified. AutoThink is not served. - LAYA text serving. LAYA runs as an external CPU Python worker, the in-repository Python MPS/CPU worker, or a native Rust/CUDA worker on Hopper.
- CUDA backend and planned Metal backend. High-performance GPU operations for NVIDIA GPUs, with a native Apple-GPU backend planned alongside it.
- Benchmark harness. Request replay, output-fidelity checks, and a CUDA comparison protocol across serving backends.
Share processing and scheduling; let each model own its execution.
The diagram shows the target architecture. Today the frontend forwards HTTP requests to separately running workers, whose handlers coordinate independent processors and executors. Native workers use shared FIFO admission and blocking dispatch per loaded executor. Processing orchestration, batch budgets, compatibility grouping and dynamic batching remain planned.
Open-Jev already packs candidates within one request for selected prefill GEMMs, with bounded groups and independent sequence state. See the native recipe and matched H200 measurements.
| Layer | Responsibility | Native target implementation |
|---|---|---|
| Rust frontend | API transport, request forwarding, and response delivery. | Rust. |
| Processing layer | Independent pre/postprocessing modules with model-specific processors for input preparation and output interpretation. | Rust CPU processing; GPU transforms use backends. |
| Scheduler / batcher | Queue admission, batch budgets, compatibility grouping, batch assembly, request bookkeeping, and result routing. | Rust host policy; GPU packing uses backends. |
| Model executors | Weights, forward passes, learned heads, device state, and kernel selection. | Rust orchestration calling backend operations. |
| CUDA backend | High-performance GPU operations for NVIDIA GPUs. | Rust bindings/dispatch and CUDA C++ kernels. |
| Metal backend (planned) | High-performance GPU operations for Apple GPUs. | Rust bindings/dispatch and Metal shaders. |
In the target design, the shared worker runtime invokes processors, schedules compatible work, calls the model executor, and routes each output back to its request. Tokenization, modality transforms, and response interpretation remain model-specific plugins, separate from the forward implementation. Models declare batch constraints; batch adapters pack inputs and unpack outputs using the executor's supported layout. A shared scheduler must not batch incompatible models or inputs, and dynamic batching requires executor support for real batches.
These are logical layers: the runtime and executor can share a worker process. Request bookkeeping belongs to the runtime; model device state belongs to the executor. Backends can optimize for their hardware without requiring identical internal implementations.
The architecture and integration contracts define processor/executor boundaries, compatibility grouping, state and buffer lifetimes, and result reconstruction. They also distinguish the native target from current single-prompt execution and CPU decision heads.
Implementation code lives under src/; recipes and documentation stay at the
repository root.
| Directory | Responsibility |
|---|---|
src/frontend/ |
Rust serving code, Python worker adapters, and the small engine interface. |
src/runtime/ |
Shared native execution admission and blocking dispatch. |
src/models/ |
Model contracts, existing worker pipelines, and model executors, including the shared Qwen3.5/3.8 prefill implementation. |
src/backends/cuda/ |
NVIDIA GPU operations and kernel integration. |
src/backends/metal/ |
Apple GPU operations and kernel integration. |
recipe/ |
Model setup instructions, launch commands, configuration examples, and example requests. |
docs/ |
Project documentation and architecture assets. |
The frontend, native runtime, the native workers, their shared Qwen3.5/3.8 prefill implementation and the Laya CUDA backend are Cargo workspace members. Each model and backend documents its implementation and process boundary.
Follow Your first decision on CPU (中文入口) for prerequisites, pinned dependencies, worker and frontend startup, readiness checks, a refund request and expected output. The recommended path uses the upstream LAYA English worker on Linux x86_64 with Python 3.12 and stable Rust; it needs no GPU or weight export.
The recorded CPU demo includes the
actual JSON and checks for choice, score and noul. Download/build/loading
time is separate from inference; no setup-duration or CPU speed claim is made.
For other hardware, see the LAYA MPS recipe or
the accelerated native Open-Jev recipe.
The frontend documentation describes transport and configuration.
The LFM2.5 recipe covers choice scoring with configurable candidate batches.
LAYA text serving uses the upstream CPU worker, the in-repository Python MPS/CPU
worker, or a native Rust/CUDA worker on Hopper. The Cua-S1 4B 0.2 text adapter
runs as a Python worker or as a native worker on CUDA. Open-Jev-27B-v1.1
and Open-Jev-9B run on the same native Rust/CUDA worker. Cua-S1 also has a
Python screenshot worker, Valen-Preview-0923 has a Python reference worker for
text and one-image state, CLM has a stub-encoder contract recipe, and OmniJev-4B
v1.1 runs on a native Rust/CUDA worker. D3
(Decision 3.0) has a Python reference worker that runs the checkpoint's own
pinned runtime for text, image and video state, and Vela 2.0 0.3B has one
that serves all five SystemOne question kinds (choice/noul/score plus
the span/set extensions) on the checkpoint's own pinned runtime.
pplx-decider-v1.1-27b has a Python reference worker for the pinned Perplexity
checkpoint. Public231 concurrency-1 latency for heads 1ba07d1, f968e576,
ffae182, 4b99b73, and fcf5e2b is recorded in the recipe:
| Model | Status |
|---|---|
| LAYA | External worker; Python worker on Apple Silicon (MPS) and CPU; CPU checkpoint reader; native Rust/CUDA worker on Hopper |
| Decider-2B v11 | Native Rust/CUDA text worker; Choice/Noul/isolated Score; opt-in batching, Graph and request-local prefixes; validation scope |
Cua-S1 4B 0.2 (text adapter) |
Python worker; native worker, CUDA; unmerged profile validated on the cumulative A800 sm_80 build, earlier merged path on sm_89 |
Cua-S1 4B 0.2 (multimodal adapter) |
Python CUDA reference and native Rust/CUDA screenshot worker; one PNG/JPEG image, choice; scoped RTX 4090 validation |
| JEMM (Qwen3.8-27B) | Native Rust/CUDA worker; text and 1–4-image frozen-corpus parity on A800 |
| Valen-Preview-0923 | Python reference worker; text state or one inline PNG/JPEG image, choice/noul/score; author-run on RTX 5060 Laptop GPU (sm_120, WSL2), owner: community |
| Open-Jev-27B-v1.1 | Native Rust/CUDA worker; eager independent text candidates; H200 validation |
| Open-Jev-9B | The same native Rust/CUDA worker; eager independent text candidates; reference comparison on sm_89 |
| StartLux-Decision v1.1 4B Instant | Native Rust/CUDA text worker; choice / noul / score; weights CC BY-NC 4.0; Instant only (no AutoThink, no 27B); GPU parity unverified; L20X latency recipe |
| autotrust/JEV-27B-VL | Experimental Rust/CUDA worker; single-question text, online inline PNG/JPEG and offline-preencoded image inputs; validation scope and limits |
| CLM-v0.1-8B | External worker with a CPU stub encoder; contract checks only, real Qwen3-8B decisions unverified by this recipe |
| LFM2.5-350M | Transformers worker; text state and independent choice questions |
| OmniJev-4B v1.1 | Native Rust/CUDA worker, eager: one image, Choice, Noul and Score; reference comparison on sm_89 |
| D3 (Decision 3.0) | Python reference worker on the checkpoint's pinned runtime; choice/noul/score over text, image and video state; author-run on A100 for text and media serving (see the integration and media PRs) |
| Vela 2.0 0.3B | Python reference worker on the checkpoint's pinned runtime; all five kinds (choice/noul/score/span/set) for text state; CPU FP32 validated (author-run, evidence in the integration PR) |
| Clef-flash (9B) and Clef (27B) | Python reference worker for text choice/noul/score; flash is the bring-up variant. Image, video and native CUDA are not served. L20X-labeled lab Public231 C1 primary is tip 594528a9: p50 164.969 ms, p95 216.755 ms, /health fla+causal_conv1d, versus 9030065 run1 p50 247.518 ms, p95 413.025 ms (Δp50 −82.549 ms, Δp95 −196.270 ms) |
| pplx-decider-v1.1-27b | Python reference worker on the pinned checkpoint; choice/noul/score text plus up to four images. Lab L20X-labeled GPUs are the H200 target. Kept Public231 primary is fcf5e2b: 231/231, p50 87.782 ms, p95 291.183 ms, grouped-value FlashInfer with q and k packed before the L2 view (p50 −1.30%, p95 −1.56% versus 4b99b73 88.942 / 295.797). 4b99b73 was p50 −4.20%, p95 −0.22% versus ffae182 92.844 / 296.458. Uncached all-sequence FlashInfer 3c72f02 is not the primary (p50 88.808 ms, p95 300.751 ms). Short-sequence FlashInfer a50b9aa missed its gate (p50 95.446 ms, p95 300.871 ms). cuDNN-first SDPA 0dcaa0ac missed its gate (p50 138.501 ms, p95 332.859 ms). Fused gate-up 8b4ee326 missed its gate (p50 102.437 ms, p95 304.970 ms). Always-on dynamic compile e36b32ed missed the both-percentiles gate (p50 97.751 ms, p95 263.124 ms; p95 improved). The 512-token split 7e3b1be missed its gate (p50 100.037 ms, p95 257.084 ms). Isolated-median compile 34ba5de missed its gate (p50 97.084 ms, p95 256.679 ms; the 136-token probe was eager). The FlashInfer state-workspace ring 17a7b120 missed its gate (p50 89.165 ms, p95 295.723 ms). Always-on FLA residual RMSNorm 3afb472 missed its gate (p50 93.935 ms, p95 274.902 ms). Length-gated FLA residual RMSNorm 705ae089 missed its gate (p50 94.604 ms, p95 274.835 ms). use_cp=auto a6f4d9a missed its gate (p50 112.294 ms, p95 299.419 ms). Grouped-value 3ecdf5a crashed in warmup (0/231, no p50 or p95). FLA fused gated RMSNorm 801ef7b missed its gate (p50 89.172 ms, p95 274.790 ms). FLA swiglu_fwd 2bbbe4e missed its gate (owner closeout p50 97.613 ms, p95 292.795 ms) and is not installed |
Supported models and hardware lists the devices and where each worker has been run.
See the GPU serving benchmark for request replay, output-fidelity checks, and the CUDA comparison protocol. The Open-Jev H200 results cover 74 single-candidate requests and a matched comparison with raw HF Transformers and OpenJev-Fast. The Open-Jev optimization notes record PR-by-PR Rust/CUDA changes and isolated A/B measurements, with figures, numerical checks and links to the separate experiments.
The current focus is the native Cua-S1 and Open-Jev CUDA workers and the serving benchmark harness. Planned work extends the shared runtime with processing orchestration, admission budgets, compatibility grouping and bounded dynamic batching with batch-capable executors, additional model engines and GPU backends including Metal, and per-model performance measurements as implementations are added and validated.
System1-Omni is open to contributions across serving, models, backends, benchmarks, and documentation. A reproducible bug report, a carefully measured benchmark, or a clearer recipe can be just as useful as a kernel optimization.
Review the contributing guide before opening a pull request: self-review the full diff, keep serving, processing, scheduling, and model execution separate according to the architecture contracts, and run the checks appropriate to your changes.
Have an idea or found a problem? Open an issue with the details, or send a pull request. For larger changes, start a discussion in an issue so we can work through the design together.
If you find system1-omni useful, give us a star on GitHub to support the project and help others discover it!
