diff --git a/README.md b/README.md index 00727b47..a1b55e87 100644 --- a/README.md +++ b/README.md @@ -169,6 +169,7 @@ pinned runtime for text, image and video state: | Model | Status | | --- | --- | | LAYA | [External worker](recipe/laya/README.md); [Python worker on Apple Silicon (MPS) and CPU](recipe/laya/apple-silicon.md); [CPU checkpoint reader](src/models/laya/README.md); [native Rust/CUDA worker on Hopper](recipe/laya/native/README.md) | +| Decider-2B v11 | [Native Rust/CUDA text worker](recipe/decider/README.md); Choice/Noul/isolated Score; opt-in batching, Graph and request-local prefixes; [validation scope](recipe/decider/validation.md) | | Cua-S1 4B 0.2 (`text` adapter) | [Python worker](recipe/cua_s1/text.md); [native worker](recipe/cua_s1/native.md), CUDA; unmerged profile validated on the cumulative A800 sm_80 build, earlier merged path on sm_89 | | Cua-S1 4B 0.2 (`multimodal` adapter) | [Python CUDA reference](src/frontend/cua_s1.py) and [native Rust/CUDA screenshot worker](recipe/cua_s1/native_screenshot.md); one PNG/JPEG image, `choice`; scoped RTX 4090 validation | | JEMM (Qwen3.8-27B) | [Native Rust/CUDA worker](recipe/jemm/README.md); [text and 1–4-image frozen-corpus parity on A800](src/models/jemm/README.md#validation) | diff --git a/docs/architecture.md b/docs/architecture.md index 943349f4..34e4bd27 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -7,15 +7,16 @@ design. Concrete input/output types follow each executor's supported layout. ## Implementation status The [Rust frontend](../src/frontend/README.md) currently forwards HTTP requests -to separately running workers. Cua-S1 and Open-Jev have native Rust/CUDA workers +to separately running workers. Cua-S1, Open-Jev and Decider have native Rust/CUDA workers that share the [Qwen3.5/3.8 executor](../src/models/qwen3_5/native/), which accepts single prompts and bounded packed prefill. Cua-S1 and Open-Jev-9B use single-prompt calls; Open-Jev-27B-v1.1 packs candidates within one request for input and gate/up GEMMs while preserving per-sequence mixers and output/down GEMM -shapes. The executor can also run prompts that share token prefixes, each prefix -once (below); no worker uses that yet. +shapes. Decider optionally packs up to four complete rows and 4096 tokens within +one request, or selects request-local shared prefixes through the same canonical +continuation path used by JEV-VL (below). Both optimizations default off. [Laya's native worker](../src/models/laya/README.md) -uses a separate Hopper CUDA backend for one complete padded request. All three +uses a separate Hopper CUDA backend for one complete padded request. These workers coordinate independent processors and executors through `prepare` → `execute` → `finish` and use the [native runtime](../src/runtime/README.md) for FIFO admission and blocking dispatch @@ -28,7 +29,9 @@ FP32 adapters and framework-reference arithmetic. It computes selected BF16 label-head logits on CUDA, then calibrates typed answers on the CPU. JEMM admits one complete text or 1–4-image request and synchronizes both model streams before releasing its resources. See its [model contract](../src/models/jemm/README.md). -Laya computes its scorer and action head on CUDA. Its fixed-shape +Decider projects selected tied-embedding rows on CUDA with BF16 output rounding +before FP32 calibration and CPU response assembly. Laya computes its scorer and +action head on CUDA. Its fixed-shape Graph captures Encoder/Decision; gather, scorer/action head and synchronized readback remain outside capture. Native Metal remains planned; Python workers retain their documented reference/serving roles. @@ -63,6 +66,7 @@ and candidate identity, usage, and response metadata outside the executor. | --- | --- | --- | --- | | Cua-S1 | One unpadded token-ID vector and option count per question, in request order. | One FP32 answer-letter logit vector per question. | Per-question softmax, choice/confidence, ordered answers, and token usage. | | Open-Jev | Token-ID vectors grouped by question, then independent candidate, in request order. | One FP32 learned scalar per candidate in the same grouping. | Add the `noul` false logit of zero, calibrate across each complete question, and restore typed answers, usage, and metadata. | +| Decider | Complete independent token-ID rows in question order; Score expands one no/yes row per level. | One BF16-projection-rounded FP32 selected-label logit vector per row. | Per-type calibration, whole-Score normalization, ordered typed answers and unique-prefix token usage. | | Laya | One padded request: token IDs, true lengths, question types and ordered option markers; at most 16 questions, 512 tokens per row and 2048 markers. | Per-question FP32 option logits and two action logits copied back after GPU heads. | Calibrate and decode ordered `choice`, `score` and `noul` answers, usage and metadata. | | JEMM | Ordered token IDs, three-axis positions, candidate counts and optional shared image patches for one complete request. | Per-question selected BF16 label logits copied back as FP32. | Apply checksum-covered text/image calibration and reconstruct ordered `choice`, `score` and `noul` answers and usage. | @@ -72,9 +76,11 @@ Open-Jev's model-specific batch adapter packs up to 16 independent candidates an execute alone. It restores question/candidate grouping before normalization. Laya batches questions within one request. Shared runtime admission precedes blocking dispatch: Cua-S1 admits one question forward at a -time; Open-Jev and Laya admit one complete request. Cua-S1's CPU letter projection +time; Open-Jev, Decider and Laya admit one complete request. Cua-S1's CPU letter projection stays outside admission; Open-Jev's scalar heads and Laya's GPU heads and -synchronized readback remain inside their request unit. Qwen model mutexes guard +synchronized readback remain inside their request unit. Decider includes every +backbone row and CUDA head projection, synchronizes both streams and retires +failed state before releasing admission. Qwen model mutexes guard mutable state. Laya's dedicated owning thread confines its non-Send CUDA state and receives admitted work over a rendezvous channel. Cancellation after dispatch retains the scheduler permit until execution completes. Laya diff --git a/docs/supported-models.md b/docs/supported-models.md index 993471e5..4afd2514 100644 --- a/docs/supported-models.md +++ b/docs/supported-models.md @@ -19,6 +19,7 @@ Models that are being added are also tracked in issues labeled [new model](https | Valen-Preview-0923 | [Python reference worker](../recipe/valen/reference.md); text state or one inline PNG/JPEG image; `choice` questions | Not supported | Author-run on RTX 5060 Laptop (sm_120, WSL2): one text and one 2×2 PNG request; input and readout parity ≤1e-6 against reference (model outputs not compared). Owner: community. | Not supported | Python 3.12, pinned Valen source/checkpoint, Qwen3.5-2B base, BF16 CUDA | | Open-Jev-27B-v1.1 | [Native Rust/CUDA worker](../recipe/open_jev/native.md) on the shared Qwen3.5/3.8 executor | Not supported | Validated on H200 (sm_90) for the [74 single-candidate workload](../recipe/open_jev/validation.md) | Not supported | Compute capability 8.0 or newer, CUDA toolkit to build, exported merged weights and trained head | | Open-Jev-9B | The same [native Rust/CUDA worker](../recipe/open_jev/native.md) | Not supported | Validated on compute capability 8.9 against the reference for [253 requests](../recipe/open_jev/validation-9b.md) | Not supported | Compute capability 8.0 or newer, CUDA toolkit to build, exported merged weights and trained head | +| Decider-2B v11 | [Native Rust/CUDA worker](../recipe/decider/README.md) | Contract checks only; no CPU inference | RTX 4090 (`sm_89`) synthetic parity corpus; [limits and provenance](../recipe/decider/validation.md) | Not supported | Pinned original BF16 checkpoint; ABI 9 library and consumers rebuilt together | | JEMM / Qwen3.8-27B | [Native Rust/CUDA worker](../recipe/jemm/README.md) | Host checks only; no CPU inference | A800 (`sm_80`) [text and 1–4-image frozen corpus](../src/models/jemm/README.md#validation) | Not supported | Pinned `jemm-native/2` export with BF16 base and unmerged FP32 LoRA, selected untied LM head, cuDNN 9 and additive reference CUDA symbols | | autotrust/JEV-27B-VL | [Experimental native worker](../recipe/jev_vl/README.md), single-question API | Contract tests only; no CPU inference | H800 frozen-corpus check; [experimental scope and limits](../recipe/jev_vl/validation.md) | Not supported | Pinned merged checkpoint, rebuilt ABI 9 CUDA library; image inputs require offline Transformers preencoding | | CLM-v0.1-8B | [External `clm-serve` recipe](../recipe/clm/README.md) with a CPU stub embeddings server | **Stub-encoder contract checks only** ([#23](https://github.com/ThinkFlowLab/system1-omni/pull/23)); not real Qwen3-8B decisions | Real encoder unverified by the merged recipe | Unverified | Python, upstream CLM and head checkpoint; a real encoder requires a separate embeddings server | diff --git a/mkdocs.yml b/mkdocs.yml index 70302d5c..ec30436b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -72,6 +72,8 @@ nav: - Open-Jev native text worker: recipe/open_jev/native.md - Open-Jev-27B H200 validation: recipe/open_jev/validation.md - Open-Jev-9B validation: recipe/open_jev/validation-9b.md + - Decider native text worker: recipe/decider/README.md + - Decider validation: recipe/decider/validation.md - JEMM native worker: recipe/jemm/README.md - JEV-27B-VL experimental worker: recipe/jev_vl/README.md - JEV-27B-VL validation status: recipe/jev_vl/validation.md diff --git a/recipe/README.md b/recipe/README.md index 7734da6a..e771de81 100644 --- a/recipe/README.md +++ b/recipe/README.md @@ -22,6 +22,8 @@ For a first real decision, follow the [complete CPU walkthrough](../docs/getting backbone and trained decision head of Open-Jev-27B-v1.1 or Open-Jev-9B, then serve with Rust and CUDA. [Open-Jev-9B validation](open_jev/validation-9b.md) compares the 9B worker with the reference. +- [Decider-2B v11 native worker](decider/README.md): pinned BF16 text decisions, + optional request-local batching, Graph and prefix reuse; [validation](decider/validation.md). - [JEMM native worker](jemm/README.md): pinned text/image decisions and export; known multi-image numerical limitation is documented in the model contract. - [JEV-27B-VL experimental worker](jev_vl/README.md): native text and preencoded diff --git a/recipe/decider/README.md b/recipe/decider/README.md new file mode 100644 index 00000000..b572586f --- /dev/null +++ b/recipe/decider/README.md @@ -0,0 +1,220 @@ +# Native Decider-2B v11 text decisions + +Run these commands from the repository root on Linux with Rust, an NVIDIA GPU, +CUDA toolkit/nvcc and cuBLASLt. Rebuild the library and every native Qwen consumer together for ABI **9**. +The existing Qwen backend targets compute capability +8.0 or newer; hardware validation is described in [validation.md](validation.md). +The worker itself uses no Python/PyTorch serving process. Python 3.10+ is used only +for downloading and optional reference validation. Keep at least 8 GB disk free for +weights and Rust build products; reference Python packages need additional space. +Device memory includes about 3.76 GB of BF16 checkpoint data plus model/head buffers +and length-dependent activation/workspace storage. + +## Download and build + +```sh +python3 recipe/decider/download_weights.py /path/to/models/decider-2b-v11 +cargo build --release --locked -p omni-decider-native -p omni-jev +NVCC=/usr/local/cuda/bin/nvcc \ + bash src/backends/cuda/qwen3_5/build.sh target/release 89 +``` + +Replace `89` with the target GPU's compute capability. The download helper pins +`Mapika/decider-2b` revision `533964dae8be954c5b5e19fa4948e48408094c1e` and checks +all five downloaded files; it never downloads unpinned remote Python code. Its +`--endpoint` option can select an accessible Hugging Face mirror; fixed hashes +remain required. Interrupted downloads leave no usable unverified final file. + +Startup independently hashes the required checkpoint/config/calibration/tokenizer +and rejects changed artifacts before loading CUDA. Use the original single +`model.safetensors`, with no `model.safetensors.index.json`; no export or LoRA merge +is needed. Keep files immutable while loading and serving. + +## Start the worker and frontend + +Worker terminal: + +```sh +DECIDER_GRAPH=0 \ +DECIDER_MODEL=/path/to/models/decider-2b-v11 \ +DECIDER_CUDA_LIB="$PWD/target/release/libqwen3_5_cuda.so" \ +DECIDER_HOST=127.0.0.1 DECIDER_PORT=8000 \ + ./target/release/omni-decider +``` + +Decider defaults to eager. `DECIDER_GRAPH=1` opts the backbone into CUDA Graph +capture/replay; only 0/1 are accepted and an unset switch is off. `CUA_S1_GRAPH` +has no effect on Decider; other workers retain their existing switch. +`DECIDER_CUDA_LIB` defaults to the library beside the binary. The worker uses CUDA +device 0; use `CUDA_VISIBLE_DEVICES` to select a physical device. + +The socket binds only after artifact validation, CUDA loading and a real warmup +request. Startup failure does not advertise readiness. In another terminal: + +```sh +curl --fail http://127.0.0.1:8000/health +OMNI_JEV_BIND=127.0.0.1:8080 \ +OMNI_JEV_BACKEND_URL=http://127.0.0.1:8000 \ + ./target/release/omni-jev +``` + +Health returns the model identity, checkpoint/reference revisions, BF16/effective execution mode +and effective per-type temperatures. A retired executor returns HTTP 503 and +`status: unavailable`. The frontend exposes its existing health behavior; see the +[frontend configuration](../../src/frontend/README.md) for its own limits/timeouts. + +## First decision + +```sh +curl --fail http://127.0.0.1:8080/v1/systemone \ + -H 'Content-Type: application/json' \ + --data-binary @recipe/decider/example-request.json +``` + +The example asks a Choice, Noul and three-level Score question about one text +state. The response has model `decider-2b-v11`, ordered typed answers and usage. +Question identities are omitted from model text. The exact output depends on the +checkpoint computation; this example is not a task-quality guarantee. + +Errors use JSON `detail`: HTTP 415 for unsupported Content-Type, 413 for a body +above 8 MiB, 422 for invalid/unsupported requests, and 503 when the model becomes +unavailable after an execution failure. Request-preparation task failures return +500. Invalid inputs do not retire the executor. Empty questions return HTTP 200 +with empty answers and zero input/output usage. No partial answers are returned. + +Supported inputs are plain text/JSON state, independent Choice (2–255 options), +Noul and isolated Score (2–10 levels). State-prefix truncation is 32,768 tokens; +complete rows can include question suffixes up to 36,864 tokens. At most 1,024 +expanded rows and 1,048,576 processed tokens are accepted per request. These bounds +do not bound all waiting requests together; runtime pending-queue limits and +cross-request batching remain separate work. + +The pinned worker requires released calibration. Images, video, chat/schema-first, +packed questions, cross-request prefix caching and quantized/CPU/Metal execution +are unsupported. Request-local prefix reuse is opt-in through `DECIDER_PREFIX=1` +or the conservative `DECIDER_PREFIX=auto` policy; both require Graph off. +CUDA Graph replay is opt-in through `DECIDER_GRAPH=1`, as described below. See the [model contract](../../src/models/decider/README.md) +for native JSON restrictions and the processing/execution boundary. + +## Validation and diagnostics + +The [2026-10-13 qualification](validation.md#current-qualification-on-2026-10-13) +records this standalone ABI 7 branch on RTX 4090: five frozen requests, 20 rows, +all eager/packed/Graph/fixed/shared fidelity controls, real retirement and capture +fallback, and selected GPU regressions. Its two timing samples per mode are +only descriptive. This integrated branch combines current main and #112 and +requires a separate ABI 9 rebuild and validation. Historical device results do +not qualify this integrated source. + +```sh +cargo fmt --all --check +cargo clippy --workspace --locked --all-targets --features omni-laya/serve -- -D warnings +cargo test --workspace --locked --features omni-laya/serve +cargo build --workspace --release --locked --features omni-laya/serve +DECIDER_MODEL=/path/to/models/decider-2b-v11 \ + cargo test --release --locked -p omni-decider-native --test contract -- --ignored +DECIDER_MODEL=/path/to/models/decider-2b-v11 \ +DECIDER_CUDA_LIB="$PWD/target/release/libqwen3_5_cuda.so" DECIDER_GRAPH=0 \ + cargo test --release --locked -p omni-decider-native --test gpu -- --ignored +DECIDER_CUDA_LIB="$PWD/target/release/libqwen3_5_cuda.so" \ + cargo test --release --locked -p omni-decider-native --lib \ + bf16_head_rounds_before_fp32_calibration -- --ignored +``` + +The CPU contract target requires only tokenizer/config; the GPU target requires the +complete pinned checkpoint. The projection test needs a built backend and a GPU. + +For exact prepared rows and raw BF16-rounded logits, the diagnostic binary reads +one JSON request per line and emits rows, logits and complete response: + +```sh +cargo build --release --locked -p omni-decider-native --example decider-run +python3 -c 'import json; print(json.dumps(json.load(open("recipe/decider/example-request.json"))))' > /tmp/decider-request.jsonl +DECIDER_GRAPH=0 ./target/release/examples/decider-run \ + /path/to/models/decider-2b-v11 "$PWD/target/release/libqwen3_5_cuda.so" \ + < /tmp/decider-request.jsonl +``` + +See [validation.md](validation.md) for full-checkpoint reference comparison, +HTTP/frontend checks, prerequisites and the recorded scope. + +## Optional request-local packing + +Set `DECIDER_BATCH_MAX_ROWS=4 DECIDER_BATCH_MAX_TOKENS=4096` on `omni-decider` +to pack independent rows within one admitted request. Rows default to 1; accepted +limits are 1–4 rows and 1–4096 packed tokens. Longer complete rows execute alone. +Malformed or out-of-range settings fail startup. These limits do not change HTTP +admission, token usage, or the maximum complete-row length. + +The selected-label head uses persistent buffers sized to the row limit and projects +each batch in one BF16 GEMM. No cross-request batching is introduced. Keep the +single-row default until numerical and performance results fit your workload. + +## Optional CUDA Graph replay + +`DECIDER_GRAPH=1` works with the request-local batch limits above. The backbone +caches at most 64 captures by the **ordered sequence-length vector**, uploads +current token IDs before replay, and clears captures after synchronizing before +scratch growth. A shape miss runs eagerly once and records the warmed layer loop; +it does not launch that capture on the already-computed residual. Label projection +and response assembly remain outside the graph. + +Health's `graph` object and the diagnostic runner's outer `graph` record report +requested/enabled mode, captures, replays, fallbacks, invalidations and cached shapes. +Decision-response fields remain unchanged. Capture failure disables Graph for the +worker lifetime and keeps the completed eager result; health then reports `eager`. +Launch/inference failures still retire the worker. Capture and startup costs must +be measured separately from warm replay. + +## Optional request-local shared prefixes (pending integration) + +This integration depends on the pending Qwen continuation/fixed-GEMM/executor +stack (#98/#99). Build this integrated branch with ABI9; older libraries are +rejected. `DECIDER_PREFIX=1 DECIDER_GRAPH=0` plans exact shared token prefixes +within the admitted request and restores attention KV, GDN recurrent state and +three convolution-history rows for every branch. Contiguous rows for one question +can also share a longer question prefix. The executor ends reusable spans on +64-token boundaries and repeats their remainder in each suffix. Every branch +retains at least one token and original row order. Buffers persist but their +contents are valid only within one call; there is no cross-request cache. + +Prefix execution uses the dependency's fixed GEMM algorithms. Set `DECIDER_FIXED=1` +with prefix/Graph off for the independent full-row control. With no usable prefix +(less than one reusable 64-token chunk), prefix mode executes independent fixed +rows. Both modes preserve the batch adapter's head grouping; backbone branches +run eagerly. Combining prefix/fixed with Graph or enabling prefix and fixed +together rejects startup. All new switches default off. Health reports effective +`shared`/`fixed` mode and cumulative `prefix.shared_requests`/`saved_tokens`. +The diagnostic outer record includes the same counters; decision output and usage +do not count execution savings differently. + +Before enabling fixed/shared execution on another GPU, follow the +[device check, failure alert and restart procedure](validation.md#deploying-fixed-and-shared-modes). +Execution failures retire the worker and return 503; automatic selection has the +same failure contract whenever it actually selects a shared plan. + +## Conservative automatic prefix selection + +`DECIDER_PREFIX=auto` is a default-off alternative to forced `DECIDER_PREFIX=1`. +The worker uses shared fixed-GEMM execution only when the existing aligned prefix +plan saves at least 4096 tokens and at least one third of the original row-token +work. Other requests use the normal independent/packed eager path, including +single-row and short/no-sharing requests. This avoids forcing short work through +the slower fixed-GEMM path. These are conservative host-policy thresholds, not a +guarantee of faster execution on every GPU or workload. + +`DECIDER_PREFIX=1` and `DECIDER_FIXED=1` retain their existing forced semantics. +Auto, forced prefix and fixed modes reject `DECIDER_GRAPH=1`; all switches default +off. Auto retains request-local state only. Input/output contracts and token usage +are unchanged, but normal and fixed GEMM algorithms can differ numerically: check +the original reference gates for both selected paths. Health reports `prefix_mode: +auto`, total shared requests/saved tokens and `auto_independent_requests`; counts +include the readiness warmup. + +```sh +DECIDER_PREFIX=auto DECIDER_GRAPH=0 \ +DECIDER_BATCH_MAX_ROWS=4 DECIDER_BATCH_MAX_TOKENS=4096 \ +DECIDER_MODEL=/path/to/models/decider-2b-v11 \ +DECIDER_CUDA_LIB="$PWD/target/release/libqwen3_5_cuda.so" \ + target/release/omni-decider +``` diff --git a/recipe/decider/example-request.json b/recipe/decider/example-request.json new file mode 100644 index 00000000..fbec40a6 --- /dev/null +++ b/recipe/decider/example-request.json @@ -0,0 +1 @@ +{"state":"A customer received a damaged item and requests a refund.","questions":{"route":{"type":"choice","instructions":"Choose the responsible team.","criteria":{"billing":"Refund and billing","shipping":"Delivery status","sales":"New purchases"}},"refund":{"type":"noul","instructions":"The customer requests a refund."},"severity":{"type":"score","instructions":"Severity of the damage","criteria":["Minor","Moderate","Severe"]}}} diff --git a/recipe/decider/validation.md b/recipe/decider/validation.md new file mode 100644 index 00000000..ed9139bf --- /dev/null +++ b/recipe/decider/validation.md @@ -0,0 +1,420 @@ +# Decider-2B v11 validation on RTX 4090 + +Historical validation date: 2026-10-07. The original sections retain their own +source, environment and ABI scope. The [2026-10-13 qualification](#current-qualification-on-2026-10-13) +below covers the repaired standalone ABI 7 branch with a new frozen synthetic +corpus. Neither campaign establishes task quality, every supported GPU or +universal bit-identical agreement with Transformers. + +The current integrated branch requires **ABI 9** and a fresh rebuild of the +CUDA library and all consumers. The historical GPU results below do not qualify +this source. Current-head fixed/shared execution, Graph replay/fallback and +reference parity remain device-validation handoffs, including both forced and +automatically selected shared paths. + +## Controls and fidelity gates + +Checkpoint/tokenizer/calibration: Mapika/decider-2b +`533964dae8be954c5b5e19fa4948e48408094c1e`. Checkpoint SHA-256: +`acaef2228b134dcdc20cad4ee79219482c927ec819aa3687b9b8a575c338817f`. +Native checkpoint/config/tokenizer checks are in `src/models/decider/native/src/checkpoint.rs`. +Reference runtime: Mapika/decider `50d0be0d7cb43d2066965ce5fa7f3fe4e489a60f`, +decider-ai 1.8.1. The comparison harness verifies fixed hashes of all five imported +reference files before running either implementation and saves them in its protocol. + +RTX 4090, sm_89, 24 GiB; driver 595.71.05, CUDA toolkit 13.0.88, Rust 1.98.1, +Python 3.12, Torch 2.14.0+cu130 and Transformers 5.17.0. The original eager baseline used one complete unpadded row at a time and ABI5. +The recorded dependent prefix campaign rebuilt the backend and consumers for ABI7; followup +controls below cover unchanged eager/Graph behavior and fixed/shared execution. The reference uses `DecisionModel.slot_logits` +with unpadded independent rows and disabled cache. Its missing causal-conv1d and +flash-linear-attention packages use Transformers' PyTorch reference fallbacks. +No optimized-reference speed comparison is claimed. + +Before execution, the harness fixes these gates: + +- Maximum absolute answer-probability and per-level fit drift: **0.02**. +- Maximum Score expectation drift: **0.1**. +- Identical Choice decisions when reference top-two probability margin is at least + **0.05**; report all lower-margin disagreements. +- Exact prepared tokens, candidate IDs/order, readout positions, usage and answer + identities. Complete responses retain all model-specific fields. + +The current helpers record validation protocol **version 2** before execution. +The three fidelity gates above are unchanged. Both native and reference responses +must have the complete typed schema, finite non-boolean numeric values, valid +ranges and the pinned decimal rounding. Confidence, certainty and `x_p_max` are +checked against their formulas using the intervals implied by four-decimal +probability rounding (half a unit is 0.00005); Score confidence accounts for modal +ties hidden by that rounding. Score expectation also receives its two-decimal +formula check. These checks do not impose a common empirical tolerance on all +auxiliary fields. + +For `n` Score levels, `fit_mass` must agree with the sum of reported level fits +within `(n + 1) * 0.00005`, accounting for rounding each fit and the sum. Its +reference drift bound is `n * 0.02 + 2 * (n + 1) * 0.00005`, derived from the +unchanged per-level gate and both response rounding allowances. The protocol saves +these methods explicitly. Historical inference records retain their original run +protocol; applying the revised validator to archived records is a CPU recheck, +not a new execution or evidence for the consolidated current head. + +## Recorded eager baseline results + +The 18-request corpus has **41 independent inference rows**. It covers Choice with +2/3/10/11/26/255 options, Score with 2/3/10 isolated levels, true/false and +criteria-only Noul, structured Unicode and long arrays, mixed questions, reversed +question order, repeated input, empty questions and long context. The longest +reference-compared row is 4,140 tokens. + +Both the initial run and the source-hash-verified rerun pass the same gates: +maximum probability/per-level-fit drift **0.0119**, maximum Score drift **0.01**, +and **zero Choice disagreements**, including lower-margin cases. Raw logit drift +is reported separately (maximum 0.125); logits are not bit-identical. These are +observed results for this corpus, not a general numerical-error bound. + +CPU opt-in tests check 12 pinned token/row/usage fixtures, every label through JT, +12 response assemblies including zero-fit Score, and state-prefix truncation. +Normal CPU tests cover selected-row ordering/padding, invalid shapes/IDs/nonfinite +weights, file content/size hashes, alternate index rejection and complete-row +validation before dispatch. A real GPU microtest checks the BF16 projection tie +`1 + 0.00390625 -> 1`, before FP32 calibration, and excludes the padded label. + +Real model regression checks verify mixed answer order, exact repeated output, +validation-error recovery and empty requests without inference. The separate native GPU boundary test executes a **36,864-token complete row** +twice, rejects the next token before execution, and then reproduces an earlier short +request exactly after scratch growth. It does not expand the reference-parity claim +beyond 4,140 tokens. + +HTTP testing passes **27 checks**: 18 direct/native diagnostic/frontend comparisons, +six invalid-input cases, the worker body limit, concurrent requests and recovery +health. The concurrent check makes 12 requests across direct and frontend paths +with four client threads; all returned bytes match the serial baseline. The first +HTTP harness run stopped on a Python variable-shadowing error after its earlier +checks; the corrected full run passes. The worker binaries were unchanged. + +The test-only CUDA wrapper lets the genuine warmup projection complete, then fails +only the second `1 x 256 x 2048` head GEMM. The worker returns 503, health becomes +unavailable, two later requests are refused, and exactly two projections are seen. +Sampled per-process GPU residency drops from **4,192 MiB to 394 MiB**, demonstrating +retirement of the model allocation; these are two lifecycle samples, not peak-memory +or performance measurements. The wrapper delegates all other operations to the +real library and is never used by normal serving. + +Cancellation semantics inherit the tested shared SerialScheduler. Decider-specific +caller cancellation and injected Rust panic are not separately hardware-tested. +CPU/Metal inference, other GPUs/precisions, dynamic batching, prefix reuse, Decider +Graph replay, model-quality datasets, throughput and cold/warm latency comparisons +are outside the original eager campaign; the followup checks below cover batching, +Graph and request-local prefix reuse. + +## Reproduce + +Build and prepare the checkpoint using [the native recipe](README.md). For the +reference environment, use Python 3.12 and the recorded Torch/Transformers versions; +install Transformers 5.17.0, tokenizers 0.22.2 and Torch 2.14.0 built for CUDA 13.0. +The native worker's preparation and serving require none of these Python packages. + +Obtain the pinned reference in a separate directory: + +```sh +git clone https://github.com/Mapika/decider.git /path/to/decider-reference +git -C /path/to/decider-reference checkout 50d0be0d7cb43d2066965ce5fa7f3fe4e489a60f +``` + +From the System1-Omni repository root, with a reserved/available NVIDIA GPU: + +```sh +cargo build --release --locked -p omni-decider-native --example decider-run +DECIDER_GRAPH=0 python tests/decider/verify_reference.py \ + --model /path/to/models/decider-2b-v11 \ + --reference /path/to/decider-reference \ + --binary "$PWD/target/release/examples/decider-run" \ + --library "$PWD/target/release/libqwen3_5_cuda.so" \ + --output /path/to/evidence/parity + +python tests/decider/verify_http.py \ + --worker "$PWD/target/release/omni-decider" \ + --frontend "$PWD/target/release/omni-jev" \ + --model /path/to/models/decider-2b-v11 \ + --library "$PWD/target/release/libqwen3_5_cuda.so" \ + --parity-output /path/to/evidence/parity \ + --output /path/to/evidence/http +``` + +The HTTP runner starts local sockets on 18110/18111, checks readiness and shuts its +processes down even on failure. It retains worker/frontend logs and complete results. +Do not run it over unrelated processes already using these ports. + +For the isolated failure test, build a wrapper beside the real library so its +runtime search path resolves the actual backend: + +```sh +gcc -shared -fPIC -Wall -Wextra -I src/backends/cuda/qwen3_5 \ + tests/decider/fail_head.c -L target/release -Wl,--no-as-needed \ + -lqwen3_5_cuda -Wl,-rpath,'$ORIGIN' -ldl \ + -o target/release/decider-fail-head.so +python tests/decider/verify_retirement.py \ + --worker "$PWD/target/release/omni-decider" \ + --model /path/to/models/decider-2b-v11 \ + --wrapper "$PWD/target/release/decider-fail-head.so" \ + --output /path/to/evidence/retirement +``` + +This runner uses port 18112 and `nvidia-smi` per-process memory reporting. The recipe +also lists the explicit CPU and real-GPU Cargo tests. Ordinary workspace tests +compile/discover their ignored cases but do not establish hardware validation. + +## Evidence + +The task evidence archive retains both parity runs, the original HTTP harness +failure and its successful rerun, native/reference rows/logits/responses, protocol, +source/binary/library hashes, environment inventory, GPU tests and retirement logs. +Generated run output and binaries are kept out of the source tree; only maintained +fixtures and reproduction helpers are committed. The implementation PR links the +immutable evidence archive and its SHA-256. All requests are synthetic and carry +no private/customer content. A video is inapplicable to this text API integration. + +## Request-local batching comparison + +The followup runner `tests/decider/verify_modes.py` accepts a frozen JSON plan with +`model`, `library`, `parity_output` (the pinned reference run above), +`repetitions: 2`, `timed_cases` and `modes`. Each mode supplies `name`, an absolute +`binary` path and an `env` mapping. It hashes binaries, library, reference records +and comparison scripts before execution; retains complete records; checks prepared +rows exactly and the same response gates. Time is the diagnostic JSON-lines +roundtrip at concurrency 1, including processing, synchronization, serialization +and IPC. Startup to the empty diagnostic response and one workload warmup are +reported/excluded separately. These measurements do not establish HTTP latency, +peak memory or production throughput. Device memory is a whole-device sample at +the end of each mode, not a peak measurement. + +On the RTX 4090 followup campaign, row limits 2 and 4 passed all 18 reference +requests. Row limit 8 failed the unchanged probability gate on structured Unicode +input (0.0208 > 0.02), so supported packing is capped at **4 rows**. The failed +campaign and narrowed-protocol rerun are retained in the PR evidence archive. +Defaults remain one row; do not extrapolate these small workloads to arbitrary +requests or hardware. Batch fault injection also verifies a two-row head failure +retires the complete worker after real startup warmup. + +## Graph checks + +`DECIDER_GRAPH=1` enables explicit backbone capture/replay. The ignored root +`tests/qwen3_5/graph.rs` test uses the pinned Decider checkpoint and compares eager +and graph hidden states exactly for ordered length vectors, token changes under +the same shape, scratch growth, replay and FIFO eviction beyond 64 shapes. Run: + +```sh +DECIDER_MODEL=/path/to/models/decider-2b-v11 \ +DECIDER_CUDA_LIB="$PWD/target/release/libqwen3_5_cuda.so" CUA_S1_GRAPH=1 \ + cargo test --release --locked -p omni-qwen3-5-native --test graph \ + ordered_shape_replay_current_ids_growth_and_bounded_cache -- --ignored --test-threads=1 +``` + +Set `require_replays: true` on each Graph mode in the comparison plan to require +actual captures and warm replays with zero fallbacks. The diagnostic runner preserves +these counters separately from the decision response. `tests/decider/fail_graph.c` +is a test-only wrapper that rejects capture startup while real CUDA eager inference +continues; `verify_graph_fallback.py` checks all pinned response cases and proves +one fallback, zero captures/replays, and effective eager execution. Build the wrapper +like `fail_head.c` above, then pass `--binary`, `--model`, `--wrapper`, +`--parity-output` and `--output` to that runner. + +## Shared-prefix controls + +The dependent prefix branch consumes the canonical continuation, fixed-GEMM and +executor implementation from Qwen PRs #98/#99 while preserving main's packed +prefill and Decider Graph statistics. The duplicate private continuation path and +its additional ABI8 entry points have been removed. Both workers use public +`PrefixState` and the same `run_window` implementation. The consolidated head +requires ABI7. At the time of this historical consolidation, current-head +GPU/reference/HTTP/performance validation was unverified. The dated qualification +below records the later selected device checks; broader HTTP and performance +coverage remains outside that campaign. Rebuild all current-branch consumers with ABI9. The unchanged default GEMM handle serves the +normal/Graph path; fixed/shared modes use a separate no-split-K handle selected +at reference M64. Model tests compare shared branches bit for bit with independent +`forward_fixed` rows, including prefixes around 64-token boundaries, growth, +repetition, reordering, invalid IDs and interleaved Graph calls. Kernel tests now +include Decider's 2048 hidden width, 16 GDN heads and 8/2 attention heads alongside +the existing 4B/9B/27B shapes. Run ignored tests serially with `CUA_S1_CUDA_LIB`, +`QWEN3_5_MODEL`, `DECIDER_MODEL` and `DECIDER_CUDA_LIB` set to the rebuilt ABI9 library +and pinned Decider checkpoint. + +For response controls, put fixed4 before shared4 in the frozen modes plan and set +`exact_control: "fixed4"` and `require_shared: true` on shared4. Both use +`DECIDER_BATCH_MAX_ROWS=4` and Graph off; fixed4 sets `DECIDER_FIXED=1`, shared4 +sets `DECIDER_PREFIX=1`. This verifies exact raw logits and complete responses +between fixed/shared as well as the original Transformers fidelity gates, and +requires actual request-local token savings. Keep normal eager4 in the plan to +measure algorithm-selection cost separately from prefix reuse. + +The cached stock Qwen3.5-4B artifact has FP32 A_log and is not a valid merged BF16 +worker export; its attempted legacy-loader check fails before inference and is +retained as unavailable coverage. The same constructor/explicit-mode regression +passes on Decider2B with the inherited CUA switch. Full Cua-S1/Open-Jev decisions +are not revalidated by the Decider campaign; they require their proper merged +exports and corresponding hardware. + +The maintained `tests/decider/data/prefix-workloads.json` adds two targeted long +shared-context requests (mixed types and ten Score levels). Generate their pinned +reference records with `verify_reference.py --cases tests/decider/data/prefix-workloads.json` +and the same model/reference/binary/library/output arguments above. Then point +`verify_modes.py` at that parity directory, time exactly those two names, and keep +two repetitions per mode. This corpus is independent of the original 18-case run; +it resolves long-prefix reuse performance and does not replace broader fidelity +checks. The fixed/shared exact-logit gate also applies to measured repetitions. + +## Historical followup measurements (diagnostic roundtrip) + +The final bounded-row campaign measured `mixed` at 25.69 ms for single rows and +15.75 ms for 4-row packing; `score_10` at 51.03→29.44 ms. Independent Graph4 measured +`mixed` at 15.80→15.12 ms relative to matched eager4, while long-context single-row +latency was 89.76→89.73 ms. Each value is the median of two warm samples on the +same RTX 4090. These are narrow synthetic diagnostic timings, not HTTP latency +or a general speedup. Startup to the empty diagnostic record was measured +separately (approximately 5.5–6.4 seconds), with actual captures/replays recorded. + +The ABI 7 shared-prefix comparison retains the fixed-algorithm control separately: + +| Workload | Normal eager4 (ms) | Independent fixed4 (ms) | Shared4 (ms) | +| --- | ---: | ---: | ---: | +| Original mixed, short state |15.73|31.27|31.51| +| Original structured Unicode |32.43|47.57|41.44| +| Targeted long mixed, 5 rows / 17683 processed tokens |414.47|484.32|144.18| +| Targeted long Score10, 10 rows / 24460 processed tokens |576.17|672.89|151.96| + +Every cell uses two warmed repetitions with raw min/max retained in the archive. +The long shared runs are bit-identical in raw candidate logits and complete +responses to their independent fixed controls, and satisfy the original reference +fidelity gates. The two long cases show 2.87× and 3.79× diagnostic-roundtrip speedup +against matched normal eager4; they do not establish task quality or a workload-wide +improvement. Short states can regress due to fixed GEMM selection and branch +launch overhead, so all optimizations remain opt-in. Shared and Graph are tested +as separate modes, not combined. End-of-mode whole-device memory samples for the +long corpus were 4626 MiB eager4, 4658 MiB fixed4 and 4746 MiB shared4; peak memory +and sustained throughput were not measured. + + +## Deploying fixed and shared modes + +Before enabling `DECIDER_FIXED=1`, `DECIDER_PREFIX=1`, or automatic shared +selection on a new GPU/toolchain, build the worker and library from the same +source and run the independent, weight-free fixed-algorithm operator check on +that device: + +```sh +CUA_S1_CUDA_LIB="$PWD/target/release/libqwen3_5_cuda.so" \ + cargo test --release --locked -p omni-qwen3-5-native --test kernels \ + fixed_gemm_rows_do_not_depend_on_m -- --ignored --exact --test-threads=1 +``` + +Require one executed test and one pass, not an empty filtered run. Record the +source/library identity, GPU, driver, CUDA/cuBLAS toolchain and complete output. +This checks the listed projection shapes and row-count controls; it does not +prove support for every possible length or replace the model fidelity checks +above. The recorded RTX 4090 result is not qualification of another device. + +The fixed algorithm is checked against each requested shape. If the backend +rejects that shape, execution returns an error instead of changing algorithms. +The admitted request returns HTTP 503 without partial answers, retires the +loaded executor, and leaves `/health` at HTTP 503 with `status: unavailable`. +Use the HTTP status and health fields as the alert condition; retain the +operation context/error code and effective prefix mode for diagnosis. + +Automatic selection avoids fixed execution only for requests that its host +policy sends to the normal independent/packed path. When it selects a shared +plan, the same backend-failure and retirement rules apply; it is not a retry +with a different algorithm after an execution failure. + +For recovery, remove the failed worker from traffic, preserve its failure +record, and restart using the existing launch command with +`DECIDER_PREFIX=0 DECIDER_FIXED=0 DECIDER_GRAPH=0`. If replacing the library, +rebuild it and all consuming binaries together at the same ABI. Restore traffic +only after real warmup, ready health and a representative request succeed. +Keep the fixed/shared modes disabled on that device until their operator and +model checks pass under the original gates. A process whose executor retired +cannot recover merely by retrying a request or changing the caller's environment. + +## Current qualification on 2026-10-13 + +The repaired standalone branch was executed from committed source +`4c7d437023c47c777afc7b82e6cb95a399468785`, exported to an isolated server checkout. +The later documentation-only qualification note does not change runtime, CUDA, +test or build inputs. At the time of that campaign, main and the independent +Decider batching/Graph branches (#111/#112) used ABI **6**, and the standalone +shared-prefix branch (#113) used ABI **7**, including `cs1_gemm_create_fixed`. +The current ABI **9** union with main requires a separate rebuild and validation; +the results here +qualify the standalone ABI 7 implementation. Rebuild each library and all its +consumers together rather than using a library from another ABI. + +The run used one NVIDIA GeForce RTX 4090 (sm_89), CUDA toolkit 13.0.88, +Rust 1.98.0, Python 3.12.3, Torch 2.12.1+cu130 and Transformers 5.17.0. +Checkpoint/tokenizer/calibration remained fixed at Mapika/decider-2b +`533964dae8be954c5b5e19fa4948e48408094c1e`; the author reference was +`50d0be0d7cb43d2066965ce5fa7f3fe4e489a60f`. All five model files and five imported +reference files were hash-verified before execution. Optional FLA, causal-conv1d +and flash-attention packages were absent, so the reference used the documented +Transformers fallbacks. The existing Python environment was unchanged. This +Torch version differs from the historical 2.14.0 campaign above. + +The predeclared corpus had **five requests, 20 independent rows and 18,394 +processed tokens**, with a longest reference-compared row of **3,541 tokens**: +short mixed Choice/Score/Noul, long shared mixed, reordered short questions, +earlier short input after long shared work, and empty questions. The original +probability/per-level-fit gate (0.02), Score gate (0.1), Choice margin gate (0.05), +exact prepared-row/usage/identity checks and version 2 typed-schema/formula/rounding +checks were fixed before device execution and were not relaxed. + +| Mode | Requests passed | Maximum probability/per-level-fit drift | Maximum Score drift | +| --- | ---: | ---: | ---: | +| Independent eager | 5 | 0.0119 | 0.01 | +| Request-local packed4 | 5 | 0.0120 | 0.01 | +| Graph with packed4 | 5 | 0.0120 | 0.01 | +| Independent fixed-GEMM control with packed4 | 5 | 0.0119 | 0.01 | +| Shared prefixes with packed4 | 5 | 0.0119 | 0.01 | + +There were no Choice disagreements in this corpus. Fixed/shared raw logits and +complete responses were equal for every stored parity, warmup and repeated +record. Actual Graph capture/replay and actual request-local prefix savings were +required and observed. Native/reference parity passed all five requests; raw +responses were independently recomputed on both local and server CPU with the +unchanged comparator. + +Four real projection-failure retirement checks passed. Each isolated worker +completed genuine warmup, failed its second projection, returned HTTP 503, +reported unavailable health, refused two later requests and retired its model +allocation. The per-process residency samples were 4,192 to 394 MiB for eager +and packed execution, 4,196 to 402 MiB for Graph, and 4,224 to 394 MiB for shared +prefixes. These are before/after lifecycle samples, not peak-memory measurements. +The capture-failure wrapper also passed all five requests with one failed +capture, zero captures/replays, permanent eager fallback and preserved fidelity. + +Seven selected real-GPU regressions passed: two BF16 projection microtests, +complete eager requests, packed mixed rows, forced shared-prefix request +isolation, ordered Graph shapes/current IDs/scratch growth/cache eviction, and +fixed-GEMM row independence. Two CPU tokenizer/response fixtures passed separately +without tensor loading. This campaign did not run the 36,864-token limit, +auto-prefix policy, full frontend HTTP/parity suite, task-quality datasets, +other GPUs or the cumulative ABI 8 integration. + +Only **two warmed diagnostic roundtrip samples per mode** were recorded for the +short mixed request. Their raw values are retained as descriptive diagnostics; +they do not establish a model speedup, HTTP latency, sustained throughput or a +performance improvement. Startup is separate from those samples. + +Reproduction uses the download/build commands in [README.md](README.md), the +pinned author reference, and the maintained helpers `verify_reference.py`, +`verify_modes.py`, `verify_retirement.py` and `verify_graph_fallback.py` under +`tests/decider/`. The maintained workload sources are `workloads()` in +`verify_reference.py` and the first case of `tests/decider/data/prefix-workloads.json`; +the frozen five-case ordering, mode settings, full raw results and a 31-path +pre-GPU inventory are preserved in the maintenance evidence archive. Generated +run output is not committed as source. Publication of a reviewer-accessible fresh +archive remains a separate maintenance step; historical links retain only their +original provenance. + +Provenance SHA-256 values: + +- Committed source archive: `4835e763d824aec2eb9f6539683c86c31d2a66c20b7cb1aa01a20e869f069781`. +- Native diagnostic binary: `936d9cd20ec5881d8d89ea7d1ce7db6d15d2f5ada885f07371a6bb5bab56f2ea`. +- Rebuilt ABI 7 library: `2e6d15a1726314440e8547986fb7cd56a45fdc5931dcd56694d78883a20dff28`. +- Frozen reference records: `965e57ba64275f6db5ec0d6597da73111309e812070149a79b14dd434b67ac36`. diff --git a/recipe/open_jev/native.md b/recipe/open_jev/native.md index 463e36c0..d78315d3 100644 --- a/recipe/open_jev/native.md +++ b/recipe/open_jev/native.md @@ -67,7 +67,8 @@ or an incomplete export. The saved limit defaults to 4096 tokens per candidate; The CUDA kernels require compute capability 8.0 or newer. The current build target below is Ada (`89`); pass your GPU's compute capability explicitly. The CUDA shared library and Rust workers must be rebuilt together for ABI -version 9, which includes vision, CUDA Graph, prefix-continuation and fixed-GEMM entry points. +version 9, which includes shared vision, CUDA Graph, canonical prefix continuation, +fixed-GEMM selection and device-feature validation. ```sh src/backends/cuda/qwen3_5/build.sh target/release 89 diff --git a/src/backends/cuda/qwen3_5/README.md b/src/backends/cuda/qwen3_5/README.md index f3807ae0..d4096f2a 100644 --- a/src/backends/cuda/qwen3_5/README.md +++ b/src/backends/cuda/qwen3_5/README.md @@ -22,7 +22,7 @@ CUDA Graph, gated attention and prefix-continuation entry points and ABI 8's `cs1_get_device` and `cs1_bf16_finite` for device feature validation, and adds the fixed-algorithm GEMM handle below. Older libraries are rejected at load. See the [Cua-S1 resident-feature recipe](../../../../recipe/cua_s1/native_vision_graph.md#gpu-resident-vision-features) -for the opt-in path, lifetime contract and regression commands. +for the opt-in device feature path, lifetime contract and regression commands. cuBLASLt's heuristic picks a GEMM algorithm per M, so a row's result can change with the number of rows in the call: a prefix and its branch, run separately, round @@ -40,24 +40,37 @@ Gated DeltaNet preparation stores converted TF32 operands in three-byte componen The shared Rust model can pack independent sequences for input and gate/up GEMMs. Output/down GEMMs retain each prompt's original shape and reduction order; attention, convolution and GDN calls remain sequence-local. Packing adds no CUDA -entry points. Open-Jev uses this path within requests; Cua-S1 keeps single-prompt calls. +entry points. Open-Jev uses this path within requests; Cua-S1 keeps single-prompt calls. Decider optionally packs complete +question rows within one admitted request. Prefix continuation retains full-attention KV, three convolution input rows, and FP32 Gated DeltaNet state at 64-token chunk boundaries. It is used by the experimental JEV-VL worker and by the shared executor's `forward_shared`, which runs prompts that share prefixes once per prefix with the fixed-algorithm GEMM handle ([#85](https://github.com/ThinkFlowLab/system1-omni/issues/85)); existing -full-prefill callers keep their own paths. +full-prefill callers keep their own paths. Fixed/shared execution rejects +unmerged adapters, reference arithmetic and FP32 GDN before allocating execution +scratch or selecting the fixed GEMM handle. ABI 6 adds mandatory copy and prefix entry points after ABI 5 native vision, -and ABI 9 adds the fixed-algorithm GEMM handle. +and the original ABI 7 branch adds the fixed-algorithm GEMM handle. ABI 9 +combines this handle with the current ABI 8 device-feature contract. Rebuild the library and **all** Qwen worker binaries together; older libraries are rejected. Current-branch GPU regression is required before release. +Decider request-local prefix execution uses the same public `PrefixState` and +`run_window` continuation path as JEV-VL, with fixed-GEMM selection for its +independent/shared controls. No separate cached-attention, convolution-history +or GDN-state ABI is required. See the [Decider recipe](../../../../recipe/decider/README.md). +The Rust loader checks ABI version before resolving required symbols, so older +libraries report the rebuild requirement. Decider Graph statistics and synchronized +stream disposal are preserved; device validation of this consolidated head remains +unverified. Historical results retain their original implementation provenance. + ## Optional JEMM reference arithmetic JEMM enables the additive `cs1_reference_*` API group after model load and before any forward. It requires cuDNN 9; build with `CUDNN_INCLUDE_DIR` and -`CUDNN_LIB_DIR`. Builds without those variables retain the current ABI-8 functions; +`CUDNN_LIB_DIR`. Builds without those variables retain the current ABI-9 functions; JEMM reports a missing-capability error rather than silently selecting older numerics. The reference path uses a per-handle GEMM reduction policy, separate normalization, diff --git a/src/models/decider/README.md b/src/models/decider/README.md new file mode 100644 index 00000000..40076ccb --- /dev/null +++ b/src/models/decider/README.md @@ -0,0 +1,94 @@ +# Decider-2B v11 native worker + +The native Rust/CUDA worker serves text `choice`, `noul` and isolated-level `score` +through `POST /v1/systemone`. Follow the [native recipe](../../../recipe/decider/README.md). +The [validation protocol](../../../recipe/decider/validation.md) distinguishes +CPU contract checks, real checkpoint inference and HTTP checks. + +## Pinned artifacts and contract + +- Model/tokenizer/config: Mapika/decider-2b + `533964dae8be954c5b5e19fa4948e48408094c1e` (released 2B v11). +- Reference: Mapika/decider `50d0be0d7cb43d2066965ce5fa7f3fe4e489a60f`, decider-ai 1.8.1. +- BF16, 24 Qwen3.5 layers, hidden width 2048, 18 Gated DeltaNet and six full-attention + layers. Output embeddings are tied; there is no adapter merge or separate trained head. +- Plain state-first, independent questions; Score levels become separate no/yes rows. + Choice supports 2–255 single-token labels A through JT; Score supports 2–10 levels. +- Per-type temperatures are Choice 1.164, Noul 1.624 and Score 1.124, read from the + verified released configuration. The worker accepts only the original calibration + artifact. The CPU library's configuration API also supports explicit calibration + overrides, but those are outside the pinned worker's supported scope. +- Preserve ordered structured state/question rendering, long-array annotations, + per-question output ordering, confidence/certainty/legend/level-fit fields and + Python-style response rounding. `usage.input_tokens` counts the common prefix once; + `usage.output_tokens` is zero. Empty questions return empty answers without execution. + +State truncation caps the tokenized `Context:` prefix at 32,768 tokens; the question +suffix is additional. Native admission bounds complete rows to 36,864 tokens, +expanded rows to 1,024, total processed row tokens to 1,048,576 and raw bodies to +8 MiB. Unsupported modes, duplicate JSON keys, depth >=128, nonfinite literals +and integers outside i64/u64 are rejected. No image/video, chat/schema-first, +packed-question execution, neutralization, quantization, CPU/Metal inference, +cross-request prefix cache is included. CUDA Graph replay and request-local prefix +reuse are optional and default off. Prefix integration is a pending integration until +Qwen PRs #98/#99 merge. + +## Ownership and execution + +`processing.rs` compiles the entire request before device work. Prepared rows own +unpadded token IDs, final-position readout, candidate IDs and original identities; +`ResponseContext` owns whole-question calibration and answer reconstruction. + +`checkpoint.rs` verifies fixed sizes and SHA-256 hashes for the checkpoint, +configuration, calibration and tokenizer before CUDA initialization. It rejects +sharded indices that could redirect the shared loader away from the verified file. +Keep artifacts immutable for the entire worker lifetime. Selected BF16 output rows +are loaded from the pinned tied input embedding; all 255 rows are retained, with a +256th zero row for aligned CUDA GEMM storage. + +`executor.rs` owns the shared Qwen model and a separate CUDA projection stream, +GEMM handle and persistent head buffers. Each complete request gets one shared +`SerialScheduler` permit. Default execution runs rows eagerly in order. Optional request-local packing +uses `DECIDER_BATCH_MAX_ROWS` (1–4, default 1) and +`DECIDER_BATCH_MAX_TOKENS` (1–4096, default 4096). Contiguous complete rows are +packed with the shared backbone `forward_batch`; each sequence resets its own +GDN/attention state. A row above the packing budget runs alone under the unchanged +complete-row limit. A persistent head projects all batch hidden rows in one GEMM. +Score levels can cross batch boundaries; response normalization still uses the +complete question. This does not combine separate requests or change admission. The BF16 selected projection uses existing backend GEMM, then converts +its BF16 outputs to FP32 before CPU temperature scaling and normalization. Padding +never participates in softmax or token accounting. Current GEMM/reduction algorithms +can differ from Transformers; numerical agreement is measured rather than assumed. + +Decider uses an explicit shared-model loader option controlled only by +`DECIDER_GRAPH`; existing workers retain their `CUA_S1_GRAPH` constructor behavior. +The backbone caches up to 64 ordered sequence-length shapes, clears them before +scratch growth, and falls back permanently to eager if capture fails. Effective +mode and cumulative capture/replay counters appear in health/diagnostic metadata. +The selected head and calibrated response remain outside the graph. + +`prefix.rs` independently plans exact request/question token prefixes from complete +prepared rows. `DECIDER_PREFIX=1` uses shared Qwen continuation and fixed GEMM +selection; `DECIDER_FIXED=1` provides independent fixed full rows for a matched +control. Short/no-sharing cases run independent fixed rows. These paths require +Graph off. The selected head keeps identical batch grouping; calibrated responses +and unique-prefix usage remain unchanged. Prefix snapshots/KV are overwritten per +request and never reused across calls. The backend/consumers require ABI9 rebuilds. + +Both streams synchronize on completion, errors and caught execution panics before +admission is released. An execution failure retires the loaded model/head and makes +health unavailable. Validation errors preserve readiness. Queued cancellation +removes the caller; after dispatch, resources and admission remain held until work +completes. FIFO waiting is serial and inherits the current runtime's unbounded +pending queue; no new queue/token scheduling policy is claimed. + +`engine.rs` assembles the processor/executor and performs a real warmup before +binding a socket. `serve.rs` handles transport, status codes and worker health. +HTTP CPU preparation uses a blocking task outside GPU admission and the model lock. +The existing Rust frontend forwards bytes to this separately running worker. + +## Tests + +All test bodies and fixtures live under root `tests/decider`. Normal Cargo tests +require neither CUDA nor downloads. Explicit checkpoint/accelerator tests are +registered and ignored until their prerequisites are supplied; see the recipe. diff --git a/src/models/decider/native/Cargo.toml b/src/models/decider/native/Cargo.toml index c11f694d..ca279132 100644 --- a/src/models/decider/native/Cargo.toml +++ b/src/models/decider/native/Cargo.toml @@ -38,6 +38,14 @@ path = "../../../../tests/decider/config.rs" name = "batching" path = "../../../../tests/decider/batching.rs" +[[test]] +name = "options" +path = "../../../../tests/decider/options.rs" + +[[test]] +name = "prefix" +path = "../../../../tests/decider/prefix.rs" + [[example]] name = "decider-run" path = "../../../../tests/decider/runner.rs" diff --git a/src/models/decider/native/src/engine.rs b/src/models/decider/native/src/engine.rs index 09ae64ad..3d89dd00 100644 --- a/src/models/decider/native/src/engine.rs +++ b/src/models/decider/native/src/engine.rs @@ -16,13 +16,49 @@ pub struct Engine { } impl Engine { pub async fn load(dir: &Path, library: &Path) -> Result { - Self::load_with_batch(dir, library, BatchLimits::from_env()?).await + let graph = crate::options::graph_env()?; + Self::load_with_modes( + dir, + library, + BatchLimits::from_env()?, + graph, + crate::options::prefix_env(graph)?, + ) + .await } pub async fn load_with_batch( dir: &Path, library: &Path, batching: BatchLimits, ) -> Result { + Self::load_with_options(dir, library, batching, false).await + } + pub async fn load_with_options( + dir: &Path, + library: &Path, + batching: BatchLimits, + graph: bool, + ) -> Result { + Self::load_with_modes( + dir, + library, + batching, + graph, + crate::prefix::PrefixMode::Off, + ) + .await + } + pub async fn load_with_modes( + dir: &Path, + library: &Path, + batching: BatchLimits, + graph: bool, + prefix_mode: crate::prefix::PrefixMode, + ) -> Result { + anyhow::ensure!( + !graph || prefix_mode == crate::prefix::PrefixMode::Off, + "shared/fixed execution requires Graph off" + ); let directory = dir.to_owned(); let (processor,checkpoint,mut metadata) = tokio::task::spawn_blocking(move || -> Result<_> { let processor = Processor::load(&directory,Limits::default())?; @@ -33,9 +69,19 @@ impl Engine { Ok((processor,checkpoint,metadata)) }).await??; let labels = processor.labels().iter().map(|label| label.id).collect(); - let executor = Executor::load(dir, library, checkpoint, labels, batching).await?; + let executor = Executor::load( + dir, + library, + checkpoint, + labels, + batching, + graph, + prefix_mode, + ) + .await?; metadata["batch_limits"] = json!({"rows":batching.max_rows(),"tokens":batching.max_tokens()}); + metadata["prefix_mode"] = json!(prefix_mode.name()); let engine = Self { processor: Arc::new(processor), executor, @@ -56,6 +102,19 @@ impl Engine { } pub fn health(&self) -> Value { let mut metadata = self.metadata.clone(); + let stats = self.executor.graph_stats(); + metadata["execution"] = json!(if !self.executor.is_ready() { + "unavailable" + } else if stats.enabled { + "graph" + } else { + "eager" + }); + metadata["graph"] = json!(stats); + metadata["prefix"] = json!(self.executor.prefix_stats()); + if self.executor.is_ready() && metadata["prefix_mode"] != "off" { + metadata["execution"] = metadata["prefix_mode"].clone(); + } metadata["status"] = json!(if self.executor.is_ready() { "ready" } else { diff --git a/src/models/decider/native/src/executor.rs b/src/models/decider/native/src/executor.rs index bd832fce..5d73735b 100644 --- a/src/models/decider/native/src/executor.rs +++ b/src/models/decider/native/src/executor.rs @@ -3,12 +3,13 @@ use crate::{ Limits, RowInput, batching::BatchLimits, checkpoint::{Checkpoint, HIDDEN, LABELS, PADDED_LABELS, VOCAB}, + prefix::{PrefixMode, PrefixPlan, PrefixStats}, }; use anyhow::{Context, Result, ensure}; use half::bf16; use omni_qwen3_5_native::{ cuda::{self, DeviceBuffer, Stream}, - model::Model, + model::{GraphStats, Model}, }; use omni_runtime::SerialScheduler; use std::{ @@ -146,12 +147,51 @@ struct Loaded { model: Model, head: Head, batching: BatchLimits, + prefix_mode: PrefixMode, + prefix_stats: PrefixStats, } impl Loaded { fn execute(&mut self, rows: &[RowInput]) -> Result>> { cuda::set_device(0)?; let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { let lengths: Vec = rows.iter().map(|row| row.ids.len()).collect(); + let plan = match self.prefix_mode { + PrefixMode::Shared => PrefixPlan::new(rows), + PrefixMode::Auto => PrefixPlan::new(rows).filter(|plan| plan.worth_auto(rows)), + _ => None, + }; + if self.prefix_mode == PrefixMode::Fixed + || self.prefix_mode == PrefixMode::Shared + || plan.is_some() + { + let hidden = if let Some(plan) = plan { + let hidden = self + .model + .forward_shared(&plan.prompts)? + .into_iter() + .flatten() + .collect::>(); + self.prefix_stats.shared_requests += 1; + self.prefix_stats.saved_tokens += plan.saved_tokens as u64; + hidden + } else { + rows.iter() + .map(|row| self.model.forward_fixed(&row.ids)) + .collect::>>()? + }; + ensure!( + hidden.len() == rows.len(), + "shared backbone output count mismatch" + ); + let mut logits = Vec::with_capacity(rows.len()); + for range in self.batching.ranges(&lengths) { + let batch = &rows[range.clone()]; + let hidden: Vec<&[f32]> = hidden[range].iter().map(Vec::as_slice).collect(); + let counts: Vec<_> = batch.iter().map(|row| row.candidate_ids.len()).collect(); + logits.extend(self.head.project_batch(&hidden, &counts)?); + } + return Ok(logits); + } let mut logits = Vec::with_capacity(rows.len()); for range in self.batching.ranges(&lengths) { let batch = &rows[range]; @@ -171,6 +211,9 @@ impl Loaded { logits.extend(self.head.project_batch(&hidden, &counts)?); } } + if self.prefix_mode == PrefixMode::Auto { + self.prefix_stats.auto_independent_requests += 1; + } Ok(logits) })) .map_err(|_| anyhow::anyhow!("Decider execution panicked")) @@ -188,6 +231,8 @@ pub struct Executor { loaded: Arc>>, labels: Vec, ready: Arc, + graph_stats: Arc>, + prefix_stats: Arc>, } impl Executor { pub(crate) async fn load( @@ -196,28 +241,47 @@ impl Executor { checkpoint: Checkpoint, labels: Vec, batching: BatchLimits, + graph: bool, + prefix_mode: PrefixMode, ) -> Result { ensure!( - std::env::var("CUA_S1_GRAPH").as_deref() != Ok("1"), - "Decider eager execution requires CUA_S1_GRAPH unset or 0" + !graph || prefix_mode == PrefixMode::Off, + "shared/fixed execution requires Graph off" ); let (dir, library) = (dir.to_owned(), library.to_owned()); let loaded = tokio::task::spawn_blocking(move || -> Result { - let model = Model::load(&dir, &library)?; + let model = Model::load_with_graph(&dir, &library, graph)?; let head = Head::new(&checkpoint.head, batching.max_rows())?; Ok(Loaded { model, head, batching, + prefix_mode, + prefix_stats: PrefixStats::default(), }) }) .await??; + let graph_stats = Arc::new(Mutex::new(loaded.model.graph_stats())); Ok(Self { + graph_stats, + prefix_stats: Arc::new(Mutex::new(PrefixStats::default())), loaded: Arc::new(Mutex::new(Some(loaded))), labels, ready: Arc::new(AtomicBool::new(true)), }) } + pub fn graph_stats(&self) -> GraphStats { + *self + .graph_stats + .lock() + .unwrap_or_else(|error| error.into_inner()) + } + pub fn prefix_stats(&self) -> PrefixStats { + *self + .prefix_stats + .lock() + .unwrap_or_else(|error| error.into_inner()) + } pub fn is_ready(&self) -> bool { self.ready.load(Ordering::Acquire) } @@ -234,20 +298,35 @@ impl Executor { if rows.is_empty() { return Ok(Vec::new()); } - let (loaded, ready) = (self.loaded.clone(), self.ready.clone()); + let (loaded, ready, stats) = ( + self.loaded.clone(), + self.ready.clone(), + self.graph_stats.clone(), + ); + let prefix_stats = self.prefix_stats.clone(); scheduler .run(move || { let mut guard = match loaded.lock() { Ok(guard) => guard, Err(_) => { + let mut snapshot = stats.lock().unwrap_or_else(|error| error.into_inner()); + snapshot.enabled = false; + snapshot.cached_shapes = 0; ready.store(false, Ordering::Release); return Err(anyhow::anyhow!("poisoned executor")); } }; - let result = guard - .as_mut() - .context("executor unavailable")? - .execute(&rows); + let loaded = guard.as_mut().context("executor unavailable")?; + let result = loaded.execute(&rows); + *prefix_stats + .lock() + .unwrap_or_else(|error| error.into_inner()) = loaded.prefix_stats; + let mut snapshot = loaded.model.graph_stats(); + if result.is_err() { + snapshot.enabled = false; + snapshot.cached_shapes = 0; + } + *stats.lock().unwrap_or_else(|error| error.into_inner()) = snapshot; if result.is_err() { ready.store(false, Ordering::Release); // Loaded::execute synchronized both streams, including failure paths. diff --git a/src/models/decider/native/src/lib.rs b/src/models/decider/native/src/lib.rs index 52000fe8..3fd6067d 100644 --- a/src/models/decider/native/src/lib.rs +++ b/src/models/decider/native/src/lib.rs @@ -36,3 +36,7 @@ mod checkpoint; pub mod engine; pub mod executor; pub mod serve; + +pub mod options; + +pub mod prefix; diff --git a/src/models/decider/native/src/options.rs b/src/models/decider/native/src/options.rs new file mode 100644 index 00000000..8d9df440 --- /dev/null +++ b/src/models/decider/native/src/options.rs @@ -0,0 +1,62 @@ +//! Model-owned execution switches, independent from other workers' environment. +use anyhow::{Result, bail}; +pub fn graph_value(value: Option<&str>) -> Result { + match value { + None | Some("0") => Ok(false), + Some("1") => Ok(true), + Some(_) => bail!("DECIDER_GRAPH must be 0 or 1"), + } +} +pub fn graph_env() -> Result { + let value = match std::env::var("DECIDER_GRAPH") { + Ok(value) => Some(value), + Err(std::env::VarError::NotPresent) => None, + Err(error) => return Err(error.into()), + }; + graph_value(value.as_deref()) +} + +pub fn prefix_values( + prefix: Option<&str>, + fixed: Option<&str>, + graph: bool, +) -> Result { + use crate::prefix::PrefixMode; + let switch = |name, value| match value { + None | Some("0") => Ok(false), + Some("1") => Ok(true), + Some(_) => bail!("{name} must be 0 or 1"), + }; + let prefix = match prefix { + None | Some("0") => PrefixMode::Off, + Some("1") => PrefixMode::Shared, + Some("auto") => PrefixMode::Auto, + Some(_) => bail!("DECIDER_PREFIX must be 0, 1 or auto"), + }; + let fixed = switch("DECIDER_FIXED", fixed)?; + anyhow::ensure!( + !(prefix != PrefixMode::Off && fixed), + "select only one of DECIDER_PREFIX and DECIDER_FIXED" + ); + anyhow::ensure!( + !(graph && (prefix != PrefixMode::Off || fixed)), + "shared/fixed execution requires DECIDER_GRAPH=0" + ); + Ok(if prefix != PrefixMode::Off { + prefix + } else if fixed { + PrefixMode::Fixed + } else { + PrefixMode::Off + }) +} +pub fn prefix_env(graph: bool) -> Result { + let get = |name| match std::env::var(name) { + Ok(value) => Ok(Some(value)), + Err(std::env::VarError::NotPresent) => Ok(None), + Err(error) => Err(error), + }; + let prefix = get("DECIDER_PREFIX")?; + let fixed = get("DECIDER_FIXED")?; + prefix_values(prefix.as_deref(), fixed.as_deref(), graph) +} diff --git a/src/models/decider/native/src/prefix.rs b/src/models/decider/native/src/prefix.rs new file mode 100644 index 00000000..ea407cd4 --- /dev/null +++ b/src/models/decider/native/src/prefix.rs @@ -0,0 +1,90 @@ +//! Request-local exact token prefix planning; no state is cached across requests. +use crate::RowInput; +use omni_qwen3_5_native::model::{PromptGroup, SharedPrompts}; +pub struct PrefixPlan<'a> { + pub prompts: SharedPrompts<'a>, + pub saved_tokens: usize, +} +fn common(rows: &[RowInput], skip: usize) -> usize { + let first = &rows[0].ids; + let limit = rows + .iter() + .map(|row| row.ids.len().saturating_sub(1)) + .min() + .unwrap(); + (skip..limit) + .take_while(|&i| rows.iter().all(|row| row.ids[i] == first[i])) + .count() +} +impl<'a> PrefixPlan<'a> { + /// Conservative opt-in policy; savings are aligned token work, not a speed guarantee. + pub fn worth_auto(&self, rows: &[RowInput]) -> bool { + let total = rows.iter().map(|row| row.ids.len()).sum::(); + self.saved_tokens >= 4096 && self.saved_tokens >= total.div_ceil(3) + } + + pub fn new(rows: &'a [RowInput]) -> Option { + if rows.len() < 2 || rows.iter().any(|row| row.ids.is_empty()) { + return None; + } + let request = common(rows, 0); + let p = request / 64 * 64; + let mut saved_tokens = (rows.len() - 1) * p; + let mut groups = Vec::new(); + let mut start = 0; + while start < rows.len() { + let mut end = start + 1; + while end < rows.len() && rows[end].question_id == rows[start].question_id { + end += 1; + } + let group = &rows[start..end]; + let prefix = if group.len() > 1 { + common(group, request) + } else { + 0 + }; + let q = (request + prefix) / 64 * 64; + saved_tokens += (group.len() - 1) * (q - p); + groups.push(PromptGroup { + prefix: &group[0].ids[request..request + prefix], + branches: group + .iter() + .map(|row| &row.ids[request + prefix..]) + .collect(), + }); + start = end; + } + (saved_tokens > 0).then(|| Self { + prompts: SharedPrompts { + prefix: &rows[0].ids[..request], + groups, + }, + saved_tokens, + }) + } +} + +#[derive(Clone, Copy, Debug, Default, serde::Serialize)] +pub struct PrefixStats { + pub shared_requests: u64, + pub auto_independent_requests: u64, + pub saved_tokens: u64, +} +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub enum PrefixMode { + #[default] + Off, + Fixed, + Shared, + Auto, +} +impl PrefixMode { + pub fn name(self) -> &'static str { + match self { + Self::Off => "off", + Self::Fixed => "fixed", + Self::Shared => "shared", + Self::Auto => "auto", + } + } +} diff --git a/src/models/qwen3_5/native/Cargo.toml b/src/models/qwen3_5/native/Cargo.toml index a782622b..6a328e25 100644 --- a/src/models/qwen3_5/native/Cargo.toml +++ b/src/models/qwen3_5/native/Cargo.toml @@ -25,13 +25,21 @@ name = "kernels" path = "../../../../tests/qwen3_5/kernels.rs" [[test]] -name = "image_decode" -path = "../../../../tests/qwen3_5/image_decode.rs" +name = "graph" +path = "../../../../tests/qwen3_5/graph.rs" [[test]] -name = "device_features" -path = "../../../../tests/qwen3_5/device_features.rs" +name = "image_decode" +path = "../../../../tests/qwen3_5/image_decode.rs" [[test]] name = "prefix" path = "../../../../tests/qwen3_5/prefix.rs" + +[[test]] +name = "graph_loading" +path = "../../../../tests/qwen3_5/graph_loading.rs" + +[[test]] +name = "device_features" +path = "../../../../tests/qwen3_5/device_features.rs" diff --git a/src/models/qwen3_5/native/src/cuda.rs b/src/models/qwen3_5/native/src/cuda.rs index 2385a59b..e47bf6ec 100644 --- a/src/models/qwen3_5/native/src/cuda.rs +++ b/src/models/qwen3_5/native/src/cuda.rs @@ -331,14 +331,18 @@ pub fn load(path: &Path) -> Result<&'static Api> { path.display() ) })?; - let api = Api::resolve(lib)?; - // SAFETY: takes no arguments. - let abi = unsafe { (api.cs1_abi_version)() }; + // Check the version before resolving the complete interface, so older libraries + // report the rebuild requirement even when they lack newly required symbols. + // SAFETY: the version entry point has the same no-argument signature in every ABI. + let version = unsafe { lib.get:: u32>(b"cs1_abi_version\0") } + .context("CUDA library has no ABI version; rebuild it")?; + let abi = unsafe { version() }; ensure!( abi == ABI_VERSION, "{} has ABI version {abi}, this build needs {ABI_VERSION}; rebuild it", path.display() ); + let api = Api::resolve(lib)?; Ok(API.get_or_init(|| api)) } diff --git a/src/models/qwen3_5/native/src/model.rs b/src/models/qwen3_5/native/src/model.rs index 5aa60e8c..279bca89 100644 --- a/src/models/qwen3_5/native/src/model.rs +++ b/src/models/qwen3_5/native/src/model.rs @@ -588,6 +588,18 @@ impl Scratch { } } +/// Cumulative backbone capture/replay evidence, distinct from the decision response. +#[derive(Clone, Copy, Debug, Default, serde::Serialize)] +pub struct GraphStats { + pub requested: bool, + pub enabled: bool, + pub captures: u64, + pub replays: u64, + pub fallbacks: u64, + pub invalidations: u64, + pub cached_shapes: usize, +} + /// The cached device-side state of one token prefix (rows `[0, len)`) for the R2d /// hybrid path: per full-attention layer the post-prep K rows plus the pre-GEMM V /// rows, per Gated DeltaNet layer the float32 recurrent state and the three @@ -689,48 +701,46 @@ pub struct Model { lora: Option, stream: Stream, gemm: *mut c_void, - reference: Option<&'static cuda::Reference>, - fp32_gdn: Option, /// One GEMM algorithm per weight shape for `forward_shared`; created on first use. fixed_gemm: *mut c_void, + reference: Option<&'static cuda::Reference>, + fp32_gdn: Option, /// Buffers for the largest packed token count so far; grows as needed. scratch: Option, /// The request and group prefix states of `forward_shared`; grow as needed. shared: [Option; 2], /// Opt-in replay with at most 64 captures per mode, keyed by sequence lengths. graph_enabled: bool, + graph_stats: GraphStats, graphs: VecDeque<(Vec, cuda::Graph)>, mm_graphs: VecDeque<(Vec, cuda::Graph)>, - // The opt-in adapter path owns its stream; it retires after every device allocation. - _lora_stream: Option, + // The stream retires after every model allocation, including adapter resources. + _stream_owner: ModelStream, } // SAFETY: the raw pointers are device addresses and cuBLASLt handles owned by the // model; the engine runs one forward pass at a time behind a mutex. unsafe impl Send for Model {} -struct LoraStream(Stream); -impl Drop for LoraStream { +struct ModelStream(Stream); +impl Drop for ModelStream { fn drop(&mut self) { - // SAFETY: the adapter Model owns this stream, and has retired all allocations. + // SAFETY: the Model owns this stream, and has retired all allocations. unsafe { (cuda::api().cs1_stream_destroy)(self.0) }; } } impl Drop for Model { fn drop(&mut self) { - if self.lora.is_some() { - let _ = cuda::set_device(0); - let _ = cuda::synchronize(self.stream); - } + let _ = cuda::set_device(0); + let _ = cuda::synchronize(self.stream); self.graphs.clear(); self.mm_graphs.clear(); - if self.lora.is_some() { - self.scratch = None; - } + self.scratch = None; + self.shared = [None, None]; // Retire the FP32 cuBLAS context while its last stream is still valid. self.fp32_gdn = None; - // SAFETY: created by cs1_gemm_create(_fixed) and not destroyed before. + // SAFETY: both GEMM handles are owned by this model and destroyed once. unsafe { (cuda::api().cs1_gemm_destroy)(self.gemm); if !self.fixed_gemm.is_null() { @@ -743,8 +753,17 @@ impl Drop for Model { impl Model { /// Load the CUDA library and the weights. pub fn load(dir: &Path, library: &Path) -> Result { + Self::load_with_graph( + dir, + library, + std::env::var("CUA_S1_GRAPH").as_deref() == Ok("1"), + ) + } + + /// Explicit mode for model adapters; does not read another worker's switch. + pub fn load_with_graph(dir: &Path, library: &Path, graph: bool) -> Result { let cfg = Config::load(dir)?; - Self::load_checkpoint(dir, library, cfg, None) + Self::load_checkpoint(dir, library, cfg, None, graph) } /// Load an immutable FP32 PEFT language adapter without merging it into BF16 weights. @@ -758,7 +777,13 @@ impl Model { ) -> Result { let cfg = Config::load(dir)?; let adapter = lora::Checkpoint::load(&cfg, adapter_path, scale)?; - Self::load_checkpoint(dir, library, cfg, Some(adapter)) + Self::load_checkpoint( + dir, + library, + cfg, + Some(adapter), + std::env::var("CUA_S1_GRAPH").as_deref() == Ok("1"), + ) } fn load_checkpoint( @@ -766,11 +791,12 @@ impl Model { library: &Path, cfg: Config, adapter: Option, + graph: bool, ) -> Result { cuda::load(library)?; cuda::set_device(0)?; let stream = cuda::new_stream()?; - let stream_owner = adapter.as_ref().map(|_| LoraStream(stream)); + let stream_owner = ModelStream(stream); let weights = Weights::load(dir, stream)?; let lora = adapter .map(|source| source.upload(cfg.full_attention.len(), stream)) @@ -842,19 +868,31 @@ impl Model { lora, stream, gemm, + fixed_gemm: std::ptr::null_mut(), reference: None, fp32_gdn: None, - fixed_gemm: std::ptr::null_mut(), scratch: None, shared: [None, None], - graph_enabled: std::env::var("CUA_S1_GRAPH").as_deref() == Ok("1"), + graph_enabled: graph, + graph_stats: GraphStats { + requested: graph, + ..GraphStats::default() + }, graphs: VecDeque::new(), - _lora_stream: stream_owner, + _stream_owner: stream_owner, mm_graphs: VecDeque::new(), }; Ok(model) } + pub fn graph_stats(&self) -> GraphStats { + GraphStats { + enabled: self.graph_enabled, + cached_shapes: self.graphs.len() + self.mm_graphs.len(), + ..self.graph_stats + } + } + /// Cua-S1's unmerged text path: framework norm/GEMM/attention plus FP32 GDN. pub fn enable_reference_text_numerics(&mut self) -> Result<()> { ensure!( @@ -1054,8 +1092,9 @@ impl Model { fn prepare_scratch(&mut self, t: usize) -> Result<()> { cuda::set_device(0)?; if self.scratch.as_ref().is_none_or(|s| t > s.cap) { - if self.lora.is_some() { - cuda::synchronize(self.stream)?; + cuda::synchronize(self.stream)?; + if !self.graphs.is_empty() || !self.mm_graphs.is_empty() { + self.graph_stats.invalidations += 1; } self.graphs.clear(); self.mm_graphs.clear(); @@ -1110,6 +1149,7 @@ impl Model { if self.graph_enabled { if let Some((_, graph)) = self.graphs.iter().find(|(shape, _)| *shape == lengths) { graph.launch(self.stream)?; + self.graph_stats.replays += 1; if std::env::var("CUA_S1_GRAPH_TRACE").as_deref() == Ok("1") { eprintln!("CUA_S1_GRAPH mode=text event=replay tokens={t}"); } @@ -1119,7 +1159,7 @@ impl Model { self.run(s, &lengths, false)?; cuda::synchronize(self.stream)?; let captured = cuda::Graph::capture(self.stream, || self.run(s, &lengths, false)); - self.cache_graph(&lengths, false, captured); + self.cache_graph(&lengths, false, captured)?; } } else { self.run(s, &lengths, false)?; @@ -1162,6 +1202,7 @@ impl Model { .find(|(shape, _)| shape.as_slice() == [t]) { graph.launch(self.stream)?; + self.graph_stats.replays += 1; if std::env::var("CUA_S1_GRAPH_TRACE").as_deref() == Ok("1") { eprintln!("CUA_S1_GRAPH mode=multimodal event=replay tokens={t}"); } @@ -1171,7 +1212,7 @@ impl Model { self.run(s, &[t], true)?; cuda::synchronize(self.stream)?; let captured = cuda::Graph::capture(self.stream, || self.run(s, &[t], true)); - self.cache_graph(&[t], true, captured); + self.cache_graph(&[t], true, captured)?; } } else { self.run(s, &[t], true)?; @@ -1180,16 +1221,23 @@ impl Model { } /// Both modes retain the eager miss result; capture records without executing. - fn cache_graph(&mut self, lengths: &[usize], multimodal: bool, captured: Result) { + fn cache_graph( + &mut self, + lengths: &[usize], + multimodal: bool, + captured: Result, + ) -> Result<()> { let mode = if multimodal { "multimodal" } else { "text" }; match captured { Ok(graph) => { + self.graph_stats.captures += 1; let cache = if multimodal { &mut self.mm_graphs } else { &mut self.graphs }; if cache.len() == 64 { + cuda::synchronize(self.stream)?; cache.pop_front(); } cache.push_back((lengths.to_vec(), graph)); @@ -1202,11 +1250,13 @@ impl Model { eprintln!( "CUDA Graph capture failed in {mode} mode; using eager execution: {error:#}" ); + self.graph_stats.fallbacks += 1; self.graph_enabled = false; self.graphs.clear(); self.mm_graphs.clear(); } } + Ok(()) } /// Eager multimodal execution using borrowed device-resident image rows. diff --git a/src/runtime/README.md b/src/runtime/README.md index fcb2ec0f..d38a06d8 100644 --- a/src/runtime/README.md +++ b/src/runtime/README.md @@ -1,6 +1,6 @@ # Native execution runtime -`omni-runtime` provides `SerialScheduler` for the Cua-S1, Open-Jev and Laya native +`omni-runtime` provides `SerialScheduler` for the Cua-S1, Open-Jev, Decider and Laya native workers. Their engines assemble a processor, one scheduler for the loaded executor, and the executor. HTTP handlers still coordinate `prepare` → `execute` → `finish`. @@ -15,6 +15,7 @@ state. Waiting requests no longer occupy blocking threads waiting for that mutex | --- | --- | --- | | Cua-S1 | One question's unpadded Qwen forward. Release before admitting its next question. | Request preparation, CPU letter projection and response finishing. | | Open-Jev | One complete request's independent candidate forwards and CPU scalar heads. | Request preparation and calibrated response finishing. | +| Decider | One complete request's backbone rows, selected CUDA label head and synchronized readback. | Request preparation and whole-question calibrated response finishing. | | Laya | One complete padded request, including GPU scorer/action head and copyback. | Request preparation/padding and calibrated response finishing. | Models retain weights, heads, device state, scratch buffers and graph caches. @@ -35,8 +36,8 @@ This implements serial admission per executor within a worker process. Admitted concurrency is one. Queue-length limits, token budgets, compatibility grouping, shared processing orchestration and GPU batching remain planned. The Qwen executor runs single prompts, bounded packed prefill (Open-Jev-27B-v1.1's -candidates within a request) or prompts that share prefixes in one call, which no -worker uses yet; this scheduler does not pack inputs or coordinate separately +candidates within a request and Decider's complete rows) or prompts that share +prefixes in one call (Decider opt-in); this scheduler does not pack inputs or coordinate separately running workers. CPU tests cover FIFO serialization, queued cancellation, retention of admission diff --git a/tests/cua_s1/text_export.rs b/tests/cua_s1/text_export.rs index de813edb..39b917ee 100644 --- a/tests/cua_s1/text_export.rs +++ b/tests/cua_s1/text_export.rs @@ -3,6 +3,7 @@ use serde_json::json; use std::{ fs, path::PathBuf, + sync::atomic::{AtomicU64, Ordering}, time::{SystemTime, UNIX_EPOCH}, }; @@ -13,8 +14,16 @@ impl Directory { .duration_since(UNIX_EPOCH) .unwrap() .as_nanos(); - let path = - std::env::temp_dir().join(format!("cua-text-export-{}-{nonce}", std::process::id())); + Self::with_nonce(marker, nonce) + } + + fn with_nonce(marker: serde_json::Value, nonce: u128) -> Self { + static NEXT_DIRECTORY: AtomicU64 = AtomicU64::new(0); + let sequence = NEXT_DIRECTORY.fetch_add(1, Ordering::Relaxed); + let path = std::env::temp_dir().join(format!( + "cua-text-export-{}-{nonce}-{sequence}", + std::process::id() + )); fs::create_dir(&path).unwrap(); fs::write(path.join("cua_s1_export.json"), marker.to_string()).unwrap(); Self(path) @@ -60,3 +69,30 @@ async fn reference_export_rejects_changed_adapter_bytes_before_cuda_loading() { .expect("must reject changed adapter"); assert!(error.to_string().contains("adapter checksum"), "{error}"); } + +#[test] +fn text_export_fixture_directories_are_unique_with_a_shared_clock_tick() { + let barrier = std::sync::Arc::new(std::sync::Barrier::new(8)); + let nonce = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_nanos(); + let workers: Vec<_> = (0..8) + .map(|_| { + let barrier = barrier.clone(); + std::thread::spawn(move || { + barrier.wait(); + Directory::with_nonce(json!({"format":"fixture"}), nonce) + }) + }) + .collect(); + let directories: Vec<_> = workers + .into_iter() + .map(|worker| worker.join().unwrap()) + .collect(); + let unique: std::collections::HashSet<_> = directories.iter().map(|dir| &dir.0).collect(); + assert_eq!(unique.len(), directories.len()); + for directory in &directories { + assert!(directory.0.join("cua_s1_export.json").is_file()); + } +} diff --git a/tests/decider/data/README.md b/tests/decider/data/README.md new file mode 100644 index 00000000..2ca4f4ca --- /dev/null +++ b/tests/decider/data/README.md @@ -0,0 +1,32 @@ +# Pinned CPU fixtures + +`cpu.json` retains 12 request/token/row/usage fixtures and all 255 label IDs from +Mapika/decider `50d0be0d7cb43d2066965ce5fa7f3fe4e489a60f`, using Decider-2B v11 +`533964dae8be954c5b5e19fa4948e48408094c1e`. The tokenizer SHA-256 is checked by +`Processor::load`; the tests require only tokenizer and configuration files. +The large truncation fixture is replaced with generated assertions in `contract.rs`. + +`responses.json` retains 12 pinned Torch 2.14.0 CPU assemblies, including isolated +Score, 255-label uniform Choice, and zero-fit Score. Inputs are synthetic, with no +private text. Apache-2.0 reference attribution is retained in the native crate. + +Run from the repository root: + +```sh +DECIDER_MODEL=/path/to/decider cargo test --locked -p omni-decider-native \ + --test contract -- --ignored +``` + +`verify_reference.py` independently reconstructs prompts with pinned upstream +`prompt_fast`, `systemone` and `temperature`, then compares actual BF16 CUDA +checkpoint outputs and complete native responses. It saves its protocol before +execution and retains raw rows/logits/responses outside the repository fixtures. + +## Prefix workloads + +`prefix-workloads.json` is the frozen request-local prefix comparison corpus. +It is consumed by `tests/decider/verify_reference.py --cases + tests/decider/data/prefix-workloads.json` after building the registered +`decider-run` example. It supplements the default 18-case reference corpus and +requires the pinned model, reference package and real CUDA library. Generated +outputs belong outside this data directory. diff --git a/tests/decider/data/prefix-workloads.json b/tests/decider/data/prefix-workloads.json new file mode 100644 index 00000000..e6eb141d --- /dev/null +++ b/tests/decider/data/prefix-workloads.json @@ -0,0 +1,54 @@ +[ + { + "name": "long_shared_mixed", + "request": { + "state": "The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. The item is damaged. ", + "questions": { + "route": { + "instructions": "Choose the team.", + "criteria": { + "refund": "Refund and billing", + "shipping": "Shipping tracking" + } + }, + "severity": { + "type": "score", + "instructions": "Severity of the damage.", + "criteria": [ + "Minor", + "Moderate", + "Severe" + ] + }, + "refund": { + "type": "noul", + "instructions": "The customer requests a refund." + } + } + } + }, + { + "name": "long_shared_score10", + "request": { + "state": "The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. The customer requests a refund for damage. ", + "questions": { + "severity": { + "type": "score", + "instructions": "Severity of the damage.", + "criteria": [ + "0. Damage level 0", + "1. Damage level 1", + "2. Damage level 2", + "3. Damage level 3", + "4. Damage level 4", + "5. Damage level 5", + "6. Damage level 6", + "7. Damage level 7", + "8. Damage level 8", + "9. Damage level 9" + ] + } + } + } + } +] diff --git a/tests/decider/executor.rs b/tests/decider/executor.rs index c5a73532..03c077e4 100644 --- a/tests/decider/executor.rs +++ b/tests/decider/executor.rs @@ -85,3 +85,31 @@ fn batched_head_preserves_rows_counts_and_bf16_rounding() { b[0] = f32::NAN; assert!(head.project_batch(&[&a, &b], &[2, 2]).is_err()); } + +#[test] +fn graph_snapshot_does_not_wait_for_model_lock_and_survives_retirement() { + let executor = std::sync::Arc::new(Executor { + loaded: Arc::new(Mutex::new(None)), + labels: vec![], + ready: Arc::new(AtomicBool::new(false)), + prefix_stats: Arc::new(Mutex::new(PrefixStats::default())), + graph_stats: Arc::new(Mutex::new(GraphStats { + requested: true, + captures: 2, + replays: 3, + ..GraphStats::default() + })), + }); + let lock = executor.loaded.lock().unwrap(); + let copy = executor.clone(); + let (send, recv) = std::sync::mpsc::channel(); + let reader = std::thread::spawn(move || send.send(copy.graph_stats()).unwrap()); + let result = recv.recv_timeout(std::time::Duration::from_secs(1)); + drop(lock); + reader.join().unwrap(); + let stats = result.expect("health waited on the execution lock"); + assert!(stats.requested); + assert!(!stats.enabled); + assert_eq!(stats.captures, 2); + assert_eq!(stats.replays, 3); +} diff --git a/tests/decider/fail_graph.c b/tests/decider/fail_graph.c new file mode 100644 index 00000000..633a726c --- /dev/null +++ b/tests/decider/fail_graph.c @@ -0,0 +1,3 @@ +// Test-only deterministic capture failure; eager CUDA math remains real. +#include "ops.h" +int cs1_graph_begin(void* stream) { (void)stream; return 2; } diff --git a/tests/decider/gpu.rs b/tests/decider/gpu.rs index 8bec730e..07e5c1df 100644 --- a/tests/decider/gpu.rs +++ b/tests/decider/gpu.rs @@ -135,3 +135,120 @@ async fn packed_mixed_rows_preserve_order_usage_and_recovery() { ); assert_eq!(engine.predict(raw.as_bytes()).await.unwrap(), first); } + +#[tokio::test] +#[ignore = "requires pinned Decider checkpoint and ABI9 CUDA prefix execution"] +async fn shared_request_prefix_matches_fixed_rows_and_never_survives_request() { + use omni_decider_native::{batching::BatchLimits, prefix::PrefixMode}; + use serde_json::json; + let dir = std::env::var("DECIDER_MODEL").unwrap(); + let library = std::env::var("DECIDER_CUDA_LIB").unwrap(); + let fixed = Engine::load_with_modes( + Path::new(&dir), + Path::new(&library), + BatchLimits::new(4, 4096).unwrap(), + false, + PrefixMode::Fixed, + ) + .await + .unwrap(); + let shared = Engine::load_with_modes( + Path::new(&dir), + Path::new(&library), + BatchLimits::new(4, 4096).unwrap(), + false, + PrefixMode::Shared, + ) + .await + .unwrap(); + let request = |state: String| { + json!({"state":state,"questions":{"route":{"criteria":["refund","shipping"],"instructions":"Choose."},"score":{"type":"score","instructions":"Severity.","criteria":["low","medium","high"]},"last":{"type":"noul","instructions":"The item is damaged."}}}).to_string() + }; + let raw = request("The item is damaged. ".repeat(200)); + let first = shared.predict(raw.as_bytes()).await.unwrap(); + assert_eq!(first, fixed.predict(raw.as_bytes()).await.unwrap()); + assert!(shared.executor.prefix_stats().shared_requests > 0); + assert!(shared.executor.prefix_stats().saved_tokens > 0); + assert_eq!(shared.health()["execution"], "shared"); + let changed = request("The item is shipped. ".repeat(200)); + assert_eq!( + shared.predict(changed.as_bytes()).await.unwrap(), + fixed.predict(changed.as_bytes()).await.unwrap() + ); + let long = request("The item is damaged. ".repeat(1000)); + assert_eq!( + shared.predict(long.as_bytes()).await.unwrap(), + fixed.predict(long.as_bytes()).await.unwrap() + ); + assert_eq!(shared.predict(raw.as_bytes()).await.unwrap(), first); + assert!( + shared + .predict(br#"{"state":0,"questions":{"bad":{"criteria":["one"]}}}"#) + .await + .is_err() + ); + assert_eq!(shared.predict(raw.as_bytes()).await.unwrap(), first); + assert!(shared.executor.is_ready()); +} + +#[tokio::test] +#[ignore = "requires pinned Decider checkpoint and ABI9 CUDA auto-prefix execution"] +async fn auto_prefix_keeps_short_packed_outputs_and_isolates_long_shared_state() { + use omni_decider_native::{batching::BatchLimits, prefix::PrefixMode}; + use serde_json::json; + let dir = std::env::var("DECIDER_MODEL").unwrap(); + let library = std::env::var("DECIDER_CUDA_LIB").unwrap(); + let load = |mode| { + Engine::load_with_modes( + Path::new(&dir), + Path::new(&library), + BatchLimits::new(4, 4096).unwrap(), + false, + mode, + ) + }; + let eager = load(PrefixMode::Off).await.unwrap(); + let fixed = load(PrefixMode::Fixed).await.unwrap(); + let auto = load(PrefixMode::Auto).await.unwrap(); + assert_eq!(auto.health()["prefix_mode"], "auto"); + assert_eq!(auto.executor.prefix_stats().auto_independent_requests, 1); + let request = |state: String| { + json!({"state":state,"questions":{ + "route":{"criteria":["refund","shipping"],"instructions":"Choose."}, + "score":{"type":"score","instructions":"Severity.","criteria":["low","medium","high"]}, + "last":{"type":"noul","instructions":"The item is damaged."} + }}) + .to_string() + }; + let short = request("The item is damaged.".into()); + let first = auto.predict(short.as_bytes()).await.unwrap(); + assert_eq!(first, eager.predict(short.as_bytes()).await.unwrap()); + assert_eq!(auto.executor.prefix_stats().shared_requests, 0); + let long = request("The item is damaged. ".repeat(800)); + assert_eq!( + auto.predict(long.as_bytes()).await.unwrap(), + fixed.predict(long.as_bytes()).await.unwrap() + ); + assert_eq!(auto.executor.prefix_stats().shared_requests, 1); + assert!(auto.executor.prefix_stats().saved_tokens >= 4096); + let changed = request("The item is shipped. ".repeat(800)); + assert_eq!( + auto.predict(changed.as_bytes()).await.unwrap(), + fixed.predict(changed.as_bytes()).await.unwrap() + ); + assert_eq!( + auto.predict(long.as_bytes()).await.unwrap(), + fixed.predict(long.as_bytes()).await.unwrap() + ); + assert_eq!(auto.predict(short.as_bytes()).await.unwrap(), first); + assert_eq!(auto.health()["execution"], "auto"); + assert_eq!(auto.executor.prefix_stats().shared_requests, 3); + assert!(auto.executor.prefix_stats().auto_independent_requests >= 3); + assert!( + auto.predict(br#"{"state":0,"questions":{"bad":{"criteria":["one"]}}}"#) + .await + .is_err() + ); + assert_eq!(auto.predict(short.as_bytes()).await.unwrap(), first); + assert!(auto.executor.is_ready()); +} diff --git a/tests/decider/options.rs b/tests/decider/options.rs new file mode 100644 index 00000000..0eba39e3 --- /dev/null +++ b/tests/decider/options.rs @@ -0,0 +1,42 @@ +use omni_decider_native::options::graph_value; +#[test] +fn graph_switch_defaults_off_and_accepts_only_zero_or_one() { + assert!(!graph_value(None).unwrap()); + assert!(!graph_value(Some("0")).unwrap()); + assert!(graph_value(Some("1")).unwrap()); + for value in ["", "true", "false", "2", "-1", "01", " 1"] { + assert!(graph_value(Some(value)).is_err(), "{value}"); + } +} + +#[test] +fn prefix_and_fixed_controls_reject_graph_or_each_other() { + use omni_decider_native::{options::prefix_values, prefix::PrefixMode}; + assert_eq!(prefix_values(None, None, false).unwrap(), PrefixMode::Off); + assert_eq!( + prefix_values(Some("1"), None, false).unwrap(), + PrefixMode::Shared + ); + assert_eq!( + prefix_values(None, Some("1"), false).unwrap(), + PrefixMode::Fixed + ); + assert!(prefix_values(Some("1"), Some("1"), false).is_err()); + assert!(prefix_values(Some("1"), None, true).is_err()); + assert!(prefix_values(None, Some("1"), true).is_err()); + assert!(prefix_values(Some("true"), None, false).is_err()); +} + +#[test] +fn auto_prefix_mode_is_opt_in_and_rejects_fixed_or_graph() { + use omni_decider_native::{options::prefix_values, prefix::PrefixMode}; + assert_eq!( + prefix_values(Some("auto"), None, false).unwrap(), + PrefixMode::Auto + ); + assert!(prefix_values(Some("auto"), Some("1"), false).is_err()); + assert!(prefix_values(Some("auto"), None, true).is_err()); + for invalid in ["AUTO", " auto", "auto ", "2"] { + assert!(prefix_values(Some(invalid), None, false).is_err()); + } +} diff --git a/tests/decider/prefix.rs b/tests/decider/prefix.rs new file mode 100644 index 00000000..2d4970f3 --- /dev/null +++ b/tests/decider/prefix.rs @@ -0,0 +1,110 @@ +use omni_decider_native::{Kind, RowInput, prefix::PrefixPlan}; +fn row(ids: Vec, q: &str) -> RowInput { + let n = ids.len(); + RowInput { + ids, + readout_position: n - 1, + candidate_ids: vec![32, 33], + question_id: q.into(), + level_index: None, + temperature: 1., + kind: Kind::Choice, + } +} +#[test] +fn exact_token_prefixes_keep_origins_and_nonempty_branches() { + let mut a = vec![100; 130]; + a.push(101); + let mut b = vec![100; 130]; + b.push(102); + let rows = vec![row(a, "q"), row(b, "q")]; + let plan = PrefixPlan::new(&rows).unwrap(); + assert_eq!(plan.saved_tokens, 128); + assert_eq!(plan.prompts.groups.len(), 1); + for (row, branch) in rows.iter().zip(&plan.prompts.groups[0].branches) { + assert!(!branch.is_empty()); + assert_eq!( + [plan.prompts.prefix, plan.prompts.groups[0].prefix, *branch].concat(), + row.ids + ); + } +} +#[test] +fn question_prefix_and_request_prefix_are_independent_and_ordered() { + let mut a = vec![100; 64]; + a.extend(vec![101; 65]); + a.push(102); + let mut b = a.clone(); + *b.last_mut().unwrap() = 103; + let mut c = vec![100; 64]; + c.push(104); + let rows = vec![row(a, "score"), row(b, "score"), row(c, "last")]; + let plan = PrefixPlan::new(&rows).unwrap(); + assert_eq!(plan.saved_tokens, 192); + assert_eq!(plan.prompts.groups.len(), 2); + let rebuilt: Vec<_> = plan + .prompts + .groups + .iter() + .flat_map(|group| { + group + .branches + .iter() + .map(|branch| [plan.prompts.prefix, group.prefix, *branch].concat()) + }) + .collect(); + assert_eq!( + rebuilt, + rows.iter().map(|row| row.ids.clone()).collect::>() + ); +} +#[test] +fn short_unique_single_and_identical_rows_fallback_safely() { + assert!(PrefixPlan::new(&[]).is_none()); + assert!(PrefixPlan::new(&[row(vec![100; 100], "q")]).is_none()); + assert!(PrefixPlan::new(&[row(vec![100; 63], "q"), row(vec![100; 63], "q")]).is_none()); + assert!(PrefixPlan::new(&[row(vec![100; 100], "q"), row(vec![101; 100], "q")]).is_none()); + let rows = vec![row(vec![100; 65], "q"), row(vec![100; 65], "q")]; + let plan = PrefixPlan::new(&rows).unwrap(); + assert_eq!(plan.saved_tokens, 64); + assert!( + plan.prompts.groups[0] + .branches + .iter() + .all(|b| !b.is_empty()) + ); +} + +#[test] +fn auto_reuse_requires_saved_work_and_a_large_shared_fraction() { + let pair = |prefix: usize, suffix: usize| { + let mut a = vec![100; prefix]; + a.extend(vec![101; suffix]); + let mut b = vec![100; prefix]; + b.extend(vec![102; suffix]); + vec![row(a, "q"), row(b, "q")] + }; + // Exactly 4096 aligned tokens saved and at least one third of the original work. + let fits = pair(4096, 1); + assert!(PrefixPlan::new(&fits).unwrap().worth_auto(&fits)); + let too_small = pair(4032, 1); + assert!(!PrefixPlan::new(&too_small).unwrap().worth_auto(&too_small)); + let exact_fraction = pair(4096, 2048); + assert!( + PrefixPlan::new(&exact_fraction) + .unwrap() + .worth_auto(&exact_fraction) + ); + let below_fraction = pair(4096, 2049); + assert!( + !PrefixPlan::new(&below_fraction) + .unwrap() + .worth_auto(&below_fraction) + ); + let too_much_suffix = pair(4096, 8192); + assert!( + !PrefixPlan::new(&too_much_suffix) + .unwrap() + .worth_auto(&too_much_suffix) + ); +} diff --git a/tests/decider/runner.rs b/tests/decider/runner.rs index 4502c5fb..3d242273 100644 --- a/tests/decider/runner.rs +++ b/tests/decider/runner.rs @@ -28,7 +28,7 @@ async fn main() -> Result<()> { let response = prepared.context.finish(logits.clone())?; println!( "{}", - json!({"rows":rows,"logits":logits,"response":response}) + json!({"rows":rows,"logits":logits,"response":response,"graph":engine.health()["graph"],"prefix":engine.health()["prefix"]}) ); } Ok(()) diff --git a/tests/decider/test_graph_fallback.py b/tests/decider/test_graph_fallback.py new file mode 100644 index 00000000..c1c4ac8e --- /dev/null +++ b/tests/decider/test_graph_fallback.py @@ -0,0 +1,49 @@ +import importlib.util +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile +import unittest +from unittest import mock + +# The CLI's sibling imports use its script directory in normal execution. +with mock.patch.object(sys, "path", [str(Path(__file__).parent), *sys.path]): + spec = importlib.util.spec_from_file_location("decider_graph_fallback", Path(__file__).with_name("verify_graph_fallback.py")) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + + +class GraphFallbackLaunch(unittest.TestCase): + def test_launch_resets_modes_and_freezes_protocol_before_failed_process(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + parity = root / "parity" + parity.mkdir() + (parity / "protocol.json").write_text(json.dumps({"cases": []})) + (parity / "reference.jsonl").write_text("") + output = root / "output" + argv = ["verify_graph_fallback.py", "--model", "model", "--binary", "binary", + "--wrapper", "wrapper", "--parity-output", str(parity), "--output", str(output)] + inherited = {"DECIDER_GRAPH": "0", "DECIDER_PREFIX": "auto", "DECIDER_FIXED": "1", + "DECIDER_BATCH_MAX_ROWS": "1", "DECIDER_BATCH_MAX_TOKENS": "1", "CUA_S1_GRAPH": "0"} + + def launch(*args, **kwargs): + expected = {"DECIDER_GRAPH": "1", "DECIDER_PREFIX": "0", "DECIDER_FIXED": "0", + "DECIDER_BATCH_MAX_ROWS": "4", "DECIDER_BATCH_MAX_TOKENS": "4096", "CUA_S1_GRAPH": "1"} + self.assertEqual({key: kwargs["env"][key] for key in expected}, expected) + protocol = json.loads((output / "protocol.json").read_text()) + self.assertEqual(protocol["env"], expected) + self.assertEqual(protocol["validation_protocol"]["version"], 2) + raise subprocess.CalledProcessError(1, args[0]) + + with mock.patch.dict(os.environ, inherited, clear=True), mock.patch("sys.argv", argv), \ + mock.patch.object(module.subprocess, "run", side_effect=launch): + with self.assertRaises(subprocess.CalledProcessError): + module.main() + self.assertTrue((output / "protocol.json").is_file()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/decider/test_retirement.py b/tests/decider/test_retirement.py index 30e2399a..8295cb45 100644 --- a/tests/decider/test_retirement.py +++ b/tests/decider/test_retirement.py @@ -16,13 +16,14 @@ class WorkerLaunch(unittest.TestCase): - def run_cli(self, batched, inherited): + def run_cli(self, batched, inherited, options=(), retired_health=None): with tempfile.TemporaryDirectory() as directory: output = Path(directory) args = ["verify_retirement.py", "--worker", "worker", "--model", "model", "--wrapper", "wrapper.so", "--output", str(output)] if batched: args.append("--batched") + args.extend(options) def launch(*args, **kwargs): # Simulate external GPU evidence so this host-only test can @@ -36,7 +37,7 @@ def launch(*args, **kwargs): child = mock.Mock(pid=123) replies = [(503, b'{"detail":"model inference failed"}'), - (503, b'{"status":"unavailable"}'), + (503, json.dumps(retired_health or {"status": "unavailable"}).encode()), (503, b'{"detail":"model unavailable"}'), (503, b'{"detail":"model unavailable"}')] with mock.patch.dict(os.environ, inherited, clear=True), \ @@ -75,6 +76,28 @@ def test_unbatched_launch_overrides_inherited_limits_and_modes(self): "DECIDER_PREFIX": "1", "DECIDER_FIXED": "1"} self.assert_mode(self.run_cli(False, inherited), 1) + def test_explicit_graph_mode_preserves_batched_limits(self): + health = {"status": "unavailable", "execution": "unavailable", + "graph": {"requested": True, "enabled": False, "captures": 1}} + env = self.run_cli(True, {"DECIDER_PREFIX": "auto", "DECIDER_FIXED": "1"}, + options=["--graph"], retired_health=health) + self.assertEqual(env["DECIDER_GRAPH"], "1") + self.assertEqual(env["DECIDER_BATCH_MAX_ROWS"], "2") + self.assertEqual(env["DECIDER_BATCH_MAX_TOKENS"], "4096") + for name in ("CUA_S1_GRAPH", "DECIDER_PREFIX", "DECIDER_FIXED"): + self.assertEqual(env[name], "0", name) + + def test_explicit_prefix_mode_overrides_inherited_graph_and_fixed(self): + health = {"status": "unavailable", "execution": "unavailable", + "prefix_mode": "shared", "prefix": {"shared_requests": 1, "saved_tokens": 1}} + env = self.run_cli(True, {"DECIDER_GRAPH": "1", "DECIDER_FIXED": "1"}, + options=["--prefix"], retired_health=health) + self.assertEqual(env["DECIDER_PREFIX"], "1") + self.assertEqual(env["DECIDER_BATCH_MAX_ROWS"], "2") + self.assertEqual(env["DECIDER_BATCH_MAX_TOKENS"], "4096") + for name in ("CUA_S1_GRAPH", "DECIDER_GRAPH", "DECIDER_FIXED"): + self.assertEqual(env[name], "0", name) + if __name__ == "__main__": unittest.main() diff --git a/tests/decider/test_verify_http.py b/tests/decider/test_verify_http.py new file mode 100644 index 00000000..47d076ba --- /dev/null +++ b/tests/decider/test_verify_http.py @@ -0,0 +1,18 @@ +import unittest +from verify_http import concurrency_request + +class CorpusSelection(unittest.TestCase): + def test_short_custom_corpus_uses_first_nonempty_request(self): + request = {"state":"long", "questions":{"q":{}}} + self.assertIs(concurrency_request([{"name":"empty", "request":{"questions":{}}}, {"name":"long_shared", "request":request}]), request) + def test_named_mixed_case_is_preferred_regardless_of_position(self): + request = {"questions":{"mixed":{}}} + self.assertIs(concurrency_request([{"name":"other", "request":{"questions":{"q":{}}}}, {"name":"mixed", "request":request}]), request) + def test_all_empty_and_absent_corpora_are_explicit(self): + request = {"questions":{}} + self.assertIs(concurrency_request([{"name":"empty", "request":request}]), request) + with self.assertRaises(ValueError): + concurrency_request([]) + +if __name__ == "__main__": + unittest.main() diff --git a/tests/decider/verify_graph_fallback.py b/tests/decider/verify_graph_fallback.py new file mode 100644 index 00000000..d1983d1a --- /dev/null +++ b/tests/decider/verify_graph_fallback.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python3 +"""Prove genuine Decider warmup survives capture failure and stays eager.""" +import argparse +import json +import os +from pathlib import Path +import subprocess +from verify_reference import VALIDATION_PROTOCOL, compare + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + for key in ("model", "binary", "wrapper", "parity_output", "output"): + parser.add_argument("--" + key.replace("_", "-"), type=Path, required=True) + args = parser.parse_args() + args.output.mkdir(parents=True, exist_ok=False) + protocol = json.loads((args.parity_output / "protocol.json").read_text()) + reference = [json.loads(x) for x in (args.parity_output / "reference.jsonl").read_text().splitlines()] + selected_env = {"DECIDER_GRAPH": "1", "DECIDER_PREFIX": "0", "DECIDER_FIXED": "0", + "DECIDER_BATCH_MAX_ROWS": "4", "DECIDER_BATCH_MAX_TOKENS": "4096", "CUA_S1_GRAPH": "1"} + env = {**os.environ, **selected_env} + # Preserve the selected path and revised gates even if startup or capture fails. + (args.output / "protocol.json").write_text(json.dumps({**protocol, "validation_protocol": VALIDATION_PROTOCOL, + "capture_fault": "cs1_graph_begin returns2", "env": selected_env}, indent=2) + "\n") + requests = "".join(json.dumps(x["request"], ensure_ascii=False) + "\n" for x in protocol["cases"]) + with (args.output / "stderr.log").open("w") as stderr: + result = subprocess.run([str(args.binary), str(args.model), str(args.wrapper)], env=env, + input=requests, stdout=subprocess.PIPE, stderr=stderr, text=True, check=True) + (args.output / "native.jsonl").write_text(result.stdout) + records = [json.loads(x) for x in result.stdout.splitlines()] + assert len(records) == len(reference) + for ref, got in zip(reference, records): + compare(ref["response"], got["response"]) + stats = got["graph"] + assert stats["requested"] and not stats["enabled"] + assert stats["fallbacks"] == 1 and stats["captures"] == stats["replays"] == 0 + (args.output / "summary.json").write_text(json.dumps({"passed": len(records), "graph": records[-1]["graph"]}, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/tests/decider/verify_http.py b/tests/decider/verify_http.py index 595f6c64..d72da1c1 100644 --- a/tests/decider/verify_http.py +++ b/tests/decider/verify_http.py @@ -36,6 +36,15 @@ def wait_ready(base, child, log): raise TimeoutError("readiness timeout") +def concurrency_request(cases): + if not cases: + raise ValueError("empty HTTP corpus") + for case in cases: + if case["name"] == "mixed" and case["request"]["questions"]: + return case["request"] + return next((case["request"] for case in cases if case["request"]["questions"]), cases[0]["request"]) + + def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--worker", type=Path, required=True) @@ -96,7 +105,7 @@ def main(): oversized = request(worker_base, "/v1/systemone", b" " * (8 * 1024 * 1024 + 1)) assert oversized[0] == 413 checks.append({"name": "body_limit", "status": 413}) - raw = json.dumps(protocol["cases"][13]["request"]).encode() + raw = json.dumps(concurrency_request(protocol["cases"])).encode() baseline = request(worker_base, "/v1/systemone", raw) with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool: concurrent_results = list(pool.map(lambda i: request(frontend_base if i % 2 else worker_base, "/v1/systemone", raw), range(12))) diff --git a/tests/decider/verify_modes.py b/tests/decider/verify_modes.py index f488d0d9..bd2a7b85 100644 --- a/tests/decider/verify_modes.py +++ b/tests/decider/verify_modes.py @@ -39,6 +39,7 @@ def main(): "timing": "wall roundtrip: request JSON encoding/write through diagnostic stdout JSON decode; includes CPU preparation, GPU synchronization and serialization; excludes startup and one workload warmup; concurrency=1"} (args.output / "protocol.json").write_text(json.dumps(frozen, indent=2) + "\n") summaries = [] + control_records = {} for mode in plan["modes"]: directory = args.output / mode["name"] directory.mkdir() @@ -68,6 +69,10 @@ def run(request): records.write(json.dumps({"phase": "parity", "name": case["name"], "elapsed_ms": ms, "record": got}, ensure_ascii=False) + "\n") records.flush() assert got["rows"] == baseline[i]["rows"], (mode["name"], case["name"], "prepared rows") + if mode.get("exact_control"): + control = control_records[mode["exact_control"]][case["name"]] + assert got["logits"] == control["logits"] and got["response"] == control["response"], (mode["name"], case["name"], "fixed/shared exact equality") + control_records.setdefault(mode["name"], {})[case["name"]] = got checks.append({"name": case["name"], **compare(reference[i]["response"], got["response"])}) for i in selected: got, ms = run(cases[i]["request"]) @@ -80,7 +85,16 @@ def run(request): records.write(json.dumps({"phase": "measured", **sample, "record": got}, ensure_ascii=False) + "\n") records.flush() assert got["rows"] == baseline[i]["rows"], (mode["name"], cases[i]["name"], "measured rows") + if mode.get("exact_control"): + control = control_records[mode["exact_control"]][cases[i]["name"]] + assert got["logits"] == control["logits"] and got["response"] == control["response"], (mode["name"], cases[i]["name"], "measured fixed/shared equality") compare(reference[i]["response"], got["response"]) + if mode.get("require_shared"): + assert got["prefix"]["shared_requests"] > 0 and got["prefix"]["saved_tokens"] > 0 + if mode.get("require_replays"): + assert got["graph"]["enabled"] and got["graph"]["captures"] > 0 + assert got["graph"]["replays"] >= len(samples) + assert got["graph"]["fallbacks"] == 0 memory = subprocess.check_output(["nvidia-smi", "--query-gpu=memory.used", "--format=csv,noheader,nounits"], text=True).strip() summary = {"name": mode["name"], "startup_to_empty_diagnostic_ms": startup_ms, "checks": checks, "samples": samples, "device_used_mib_at_end": int(memory), diff --git a/tests/decider/verify_retirement.py b/tests/decider/verify_retirement.py index ca80bed6..9fe00b44 100644 --- a/tests/decider/verify_retirement.py +++ b/tests/decider/verify_retirement.py @@ -25,6 +25,8 @@ def main(): parser.add_argument("--wrapper", type=Path, required=True) parser.add_argument("--output", type=Path, required=True) parser.add_argument("--batched", action="store_true") + parser.add_argument("--graph", action="store_true") + parser.add_argument("--prefix", action="store_true") args = parser.parse_args() args.output.mkdir(parents=True, exist_ok=True) log = args.output / "worker.log" @@ -34,6 +36,10 @@ def main(): DECIDER_BATCH_MAX_ROWS="2" if args.batched else "1", DECIDER_BATCH_MAX_TOKENS="4096", CUA_S1_GRAPH="0", DECIDER_GRAPH="0", DECIDER_PREFIX="0", DECIDER_FIXED="0", ) + if args.graph: + environment["DECIDER_GRAPH"] = "1" + if args.prefix: + environment["DECIDER_PREFIX"] = "1" with log.open("w") as output: child = subprocess.Popen([str(args.worker)], env=environment, stdout=output, stderr=subprocess.STDOUT) try: @@ -44,10 +50,21 @@ def main(): raw = b'{"state":"The item is damaged.","questions":{"q":{"instructions":"Choose a category.","criteria":["refund","shipping"]}}}' if args.batched: raw = b'{"state":"The item is damaged.","questions":{"q":{"type":"score","instructions":"Severity.","criteria":["low","high"]}}}' + if args.prefix: + raw = json.dumps({"state": "The item is damaged. " * 200, "questions": {"q": {"type":"score", "instructions":"Severity.", "criteria":["low","high"]}}}).encode() status, body = request(base, "/v1/systemone", raw) assert status == 503 and json.loads(body)["detail"] == "model inference failed" status, body = request(base, "/health") - assert status == 503 and json.loads(body)["status"] == "unavailable" + retired_health = json.loads(body) + assert status == 503 and retired_health["status"] == "unavailable" + if args.graph: + assert retired_health["execution"] == "unavailable" + assert retired_health["graph"]["requested"] and not retired_health["graph"]["enabled"] + assert retired_health["graph"]["captures"] > 0 + if args.prefix: + assert retired_health["execution"] == "unavailable" + assert retired_health["prefix_mode"] == "shared" + assert retired_health["prefix"]["shared_requests"] == 1 and retired_health["prefix"]["saved_tokens"] > 0 for _ in range(2): status, body = request(base, "/v1/systemone", raw) assert status == 503 and json.loads(body)["detail"] == "model unavailable" @@ -60,7 +77,7 @@ def main(): if args.batched: assert "Decider test projection 2 rows 2" in text result = {"passed": True, "projection_calls": 2, "health_after": "unavailable", "response_status": 503, - "sampled_before_mib": before, "sampled_after_mib": after, "later_requests_refused": 2} + "sampled_before_mib": before, "sampled_after_mib": after, "later_requests_refused": 2, "retired_health": retired_health} (args.output / "summary.json").write_text(json.dumps(result, indent=2) + "\n") print(json.dumps(result), flush=True) finally: diff --git a/tests/qwen3_5/graph.rs b/tests/qwen3_5/graph.rs new file mode 100644 index 00000000..353496b7 --- /dev/null +++ b/tests/qwen3_5/graph.rs @@ -0,0 +1,64 @@ +//! Full-checkpoint graph lifetime, shape and current-token checks. +use omni_qwen3_5_native::model::Model; +use std::path::Path; + +#[test] +#[ignore = "requires Decider-2B checkpoint and NVIDIA CUDA"] +fn ordered_shape_replay_current_ids_growth_and_bounded_cache() { + let dir = std::env::var("DECIDER_MODEL").unwrap(); + let library = std::env::var("DECIDER_CUDA_LIB").unwrap(); + let mut eager = Model::load_with_graph(Path::new(&dir), Path::new(&library), false).unwrap(); + assert!(!eager.graph_stats().requested); + let mut graph = Model::load_with_graph(Path::new(&dir), Path::new(&library), true).unwrap(); + let a = vec![100; 64]; + let b = vec![101; 65]; + let first = graph.forward_batch(&[&a, &b]).unwrap(); + assert_eq!(first, eager.forward_batch(&[&a, &b]).unwrap()); + assert_eq!(graph.graph_stats().captures, 1); + assert_eq!(graph.forward_batch(&[&a, &b]).unwrap(), first); + assert_eq!(graph.graph_stats().replays, 1); + assert_eq!( + graph.forward_batch(&[&b, &a]).unwrap(), + eager.forward_batch(&[&b, &a]).unwrap() + ); + assert_eq!(graph.graph_stats().captures, 2); // same total, different ordered vector + let c = vec![102; 64]; + assert_eq!( + graph.forward_batch(&[&c, &b]).unwrap(), + eager.forward_batch(&[&c, &b]).unwrap() + ); + assert_eq!(graph.graph_stats().replays, 2); // fresh IDs uploaded before replay + let long = vec![100; 2049]; + assert_eq!(graph.forward(&long).unwrap(), eager.forward(&long).unwrap()); + assert_eq!(graph.graph_stats().invalidations, 1); + assert_eq!(graph.graph_stats().cached_shapes, 1); + assert_eq!(graph.forward_batch(&[&a, &b]).unwrap(), first); + for n in 1..=65 { + let ids = vec![100; n]; + graph.forward(&ids).unwrap(); + } + assert_eq!(graph.graph_stats().cached_shapes, 64); + let captures = graph.graph_stats().captures; + graph.forward(&[100]).unwrap(); + assert_eq!(graph.graph_stats().captures, captures + 1); // FIFO eviction + assert_eq!(graph.graph_stats().cached_shapes, 64); + assert_eq!(graph.graph_stats().fallbacks, 0); + graph.synchronize().unwrap(); +} + +#[test] +#[ignore = "requires Qwen3.5 checkpoint, CUA_S1_CUDA_LIB and NVIDIA CUDA"] +fn legacy_loader_retains_cua_graph_switch() { + let dir = std::env::var("QWEN3_5_MODEL").unwrap(); + let library = std::env::var("CUA_S1_CUDA_LIB").unwrap(); + let enabled = std::env::var("CUA_S1_GRAPH").as_deref() == Ok("1"); + let mut legacy = Model::load(Path::new(&dir), Path::new(&library)).unwrap(); + let mut explicit = + Model::load_with_graph(Path::new(&dir), Path::new(&library), enabled).unwrap(); + let ids = vec![100; 128]; + let first = legacy.forward(&ids).unwrap(); + assert_eq!(first, explicit.forward(&ids).unwrap()); + assert_eq!(legacy.forward(&ids).unwrap(), first); + assert_eq!(legacy.graph_stats().requested, enabled); + assert_eq!(legacy.graph_stats().replays, u64::from(enabled)); +} diff --git a/tests/qwen3_5/graph_loading.rs b/tests/qwen3_5/graph_loading.rs new file mode 100644 index 00000000..1f7dd249 --- /dev/null +++ b/tests/qwen3_5/graph_loading.rs @@ -0,0 +1,204 @@ +//! Real model constructors through a host boundary: no checkpoint download or GPU. +#![cfg(unix)] +use omni_qwen3_5_native::model::{Model, SharedPrompts}; +use safetensors::{Dtype, tensor::TensorView}; +use std::{path::Path, process::Command}; + +fn write_tensors(path: &Path, specs: &[(&str, Vec, Dtype)]) { + let storage: Vec> = specs + .iter() + .map(|(_, shape, dtype)| vec![0; shape.iter().product::() * dtype.bitsize() / 8]) + .collect(); + let views: Vec<_> = specs + .iter() + .zip(&storage) + .map(|((name, shape, dtype), data)| { + (*name, TensorView::new(*dtype, shape.clone(), data).unwrap()) + }) + .collect(); + std::fs::write(path, safetensors::tensor::serialize(views, None).unwrap()).unwrap(); +} + +#[test] +fn explicit_graph_mode_and_adapter_models_retire_the_stream_once_after_allocations() { + let directory = std::env::temp_dir().join(format!("qwen-loading-{}", std::process::id())); + std::fs::create_dir_all(&directory).unwrap(); + let config = serde_json::json!({ + "hidden_size":8,"intermediate_size":16,"rms_norm_eps":1e-6, + "num_attention_heads":1,"num_key_value_heads":1,"head_dim":256, + "max_position_embeddings":64,"linear_num_key_heads":1, + "linear_num_value_heads":1,"linear_key_head_dim":128,"linear_value_head_dim":128, + "layer_types":["full_attention"],"num_hidden_layers":1,"hidden_act":"silu", + "linear_conv_kernel_dim":4,"partial_rotary_factor":0.25,"rope_theta":10000.0 + }); + std::fs::write(directory.join("config.json"), config.to_string()).unwrap(); + let bf16 = Dtype::BF16; + write_tensors( + &directory.join("model.safetensors"), + &[ + ("embed_tokens.weight", vec![4, 8], bf16), + ("norm.weight", vec![8], bf16), + ("layers.0.input_layernorm.weight", vec![8], bf16), + ("layers.0.post_attention_layernorm.weight", vec![8], bf16), + ("layers.0.self_attn.q_proj.weight", vec![512, 8], bf16), + ("layers.0.self_attn.k_proj.weight", vec![256, 8], bf16), + ("layers.0.self_attn.v_proj.weight", vec![256, 8], bf16), + ("layers.0.self_attn.o_proj.weight", vec![8, 256], bf16), + ("layers.0.self_attn.q_norm.weight", vec![256], bf16), + ("layers.0.self_attn.k_norm.weight", vec![256], bf16), + ("layers.0.mlp.gate_proj.weight", vec![16, 8], bf16), + ("layers.0.mlp.up_proj.weight", vec![16, 8], bf16), + ("layers.0.mlp.down_proj.weight", vec![8, 16], bf16), + ], + ); + let adapter = directory.join("adapter.safetensors"); + write_tensors( + &adapter, + &[ + ( + "base_model.model.model.language_model.layers.0.mlp.down_proj.lora_A.weight", + vec![1, 16], + Dtype::F32, + ), + ( + "base_model.model.model.language_model.layers.0.mlp.down_proj.lora_B.weight", + vec![8, 1], + Dtype::F32, + ), + ], + ); + let cuda = include_str!("../../src/models/qwen3_5/native/src/cuda.rs"); + let abi = cuda + .lines() + .find(|l| l.starts_with("const ABI_VERSION:")) + .unwrap() + .split('=') + .nth(1) + .unwrap() + .trim() + .trim_end_matches(';'); + let mut fake = include_str!("graph_loading_fake.c").to_owned(); + let implemented = [ + "cs1_abi_version", + "cs1_error_string", + "cs1_get_device", + "cs1_set_device", + "cs1_malloc", + "cs1_free", + "cs1_stream_create", + "cs1_stream_sync", + "cs1_stream_destroy", + "cs1_upload", + "cs1_gemm_create", + "cs1_gemm_create_fixed", + "cs1_gemm_destroy", + ]; + let api = cuda + .split("api! {") + .nth(1) + .unwrap() + .split("\n}\n") + .next() + .unwrap(); + for line in api.lines().map(str::trim).filter(|l| l.starts_with("cs1_")) { + let name = line.split('(').next().unwrap(); + if !implemented.contains(&name) { + fake.push_str(&format!("\nint {name}(void) {{ return 77; }}\n")); + } + } + let source = directory.join("fake.c"); + let library = directory.join(if cfg!(target_os = "macos") { + "fake.dylib" + } else { + "fake.so" + }); + std::fs::write(&source, fake).unwrap(); + let output = Command::new("cc") + .args(["-shared", "-fPIC"]) + .arg(format!("-DLOADING_ABI={abi}")) + .arg(&source) + .arg("-o") + .arg(&library) + .output() + .unwrap(); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + // SAFETY: the library is retained until all model values have been destroyed. + let metrics = unsafe { libloading::Library::new(&library).unwrap() }; + let metric = + |name: &[u8]| unsafe { metrics.get:: i32>(name).unwrap()() }; + for graph in [false, true] { + let mut model = Model::load_with_graph(&directory, &library, graph).unwrap(); + assert_eq!(model.graph_stats().requested, graph); + assert_eq!(model.graph_stats().enabled, graph); + // The fake has no arithmetic kernels. The fixed handle and scratch are + // created before this deliberate dispatch error and must retire safely. + assert!(model.forward_fixed(&[0]).is_err()); + assert_eq!(metric(b"loading_gemms\0"), 2); + drop(model); + assert_eq!(metric(b"loading_streams\0"), metric(b"loading_destroyed\0")); + assert_eq!(metric(b"loading_live\0"), 0); + assert_eq!(metric(b"loading_gemms\0"), 0); + assert_eq!( + metric(b"loading_early\0"), + 0, + "stream destroyed before model allocations" + ); + } + let mut model = Model::load_with_lora(&directory, &library, &adapter, 2.0).unwrap(); + assert_eq!( + model.graph_stats().requested, + std::env::var("CUA_S1_GRAPH").as_deref() == Ok("1") + ); + let allocations = metric(b"loading_live\0"); + let error = model.forward_fixed(&[0]).unwrap_err(); + assert!(error.to_string().contains("only merged weights")); + let error = model + .forward_shared(&SharedPrompts { + prefix: &[], + groups: vec![], + }) + .unwrap_err(); + assert!(error.to_string().contains("only merged weights")); + assert_eq!( + metric(b"loading_live\0"), + allocations, + "reject before scratch allocation" + ); + assert_eq!( + metric(b"loading_gemms\0"), + 1, + "reject before fixed handle creation" + ); + drop(model); + assert_eq!(metric(b"loading_streams\0"), 3); + assert_eq!( + metric(b"loading_destroyed\0"), + 3, + "each owned stream retires exactly once" + ); + assert_eq!(metric(b"loading_live\0"), 0); + assert_eq!(metric(b"loading_early\0"), 0); + // This failure occurs after the base arena and FP32 adapter have been uploaded. + write_tensors( + &directory.join("model.safetensors"), + &[("embed_tokens.weight", vec![4, 8], bf16)], + ); + assert!(Model::load_with_lora(&directory, &library, &adapter, 2.0).is_err()); + assert_eq!(metric(b"loading_streams\0"), 4); + assert_eq!(metric(b"loading_destroyed\0"), 4); + assert_eq!(metric(b"loading_live\0"), 0); + assert_eq!(metric(b"loading_early\0"), 0); + std::fs::remove_file(directory.join("model.safetensors")).unwrap(); + assert!(Model::load_with_graph(&directory, &library, true).is_err()); + assert_eq!(metric(b"loading_streams\0"), 5); + assert_eq!( + metric(b"loading_destroyed\0"), + 5, + "failed construction retires its stream" + ); + std::fs::remove_dir_all(directory).unwrap(); +} diff --git a/tests/qwen3_5/graph_loading_fake.c b/tests/qwen3_5/graph_loading_fake.c new file mode 100644 index 00000000..ec22a5ee --- /dev/null +++ b/tests/qwen3_5/graph_loading_fake.c @@ -0,0 +1,23 @@ +// Host-only CUDA boundary for model loading and stream ownership; no kernels run. +#include +#include +#include +static int live, streams, destroyed, early, gemms; +int loading_live(void) { return live; } +int loading_streams(void) { return streams; } +int loading_destroyed(void) { return destroyed; } +int loading_early(void) { return early; } +int loading_gemms(void) { return gemms; } +uint32_t cs1_abi_version(void) { return LOADING_ABI; } +const char* cs1_error_string(int code) { (void)code; return "unexpected host-test CUDA call"; } +int cs1_get_device(int* device) { *device=0; return 0; } +int cs1_set_device(int device) { return device == 0 ? 0 : 1; } +int cs1_malloc(void** p, size_t n) { *p=malloc(n); if (!*p) return 1; ++live; return 0; } +int cs1_free(void* p) { --live; free(p); return 0; } +int cs1_stream_create(void** p) { *p=(void*)1; ++streams; return 0; } +int cs1_stream_sync(void* stream) { (void)stream; return 0; } +int cs1_stream_destroy(void* stream) { (void)stream; ++destroyed; if (live || gemms) ++early; return 0; } +int cs1_upload(void* dst, const void* src, size_t n, void* stream) { (void)stream; memcpy(dst,src,n); return 0; } +void* cs1_gemm_create(size_t n) { (void)n; ++gemms; return (void*)2; } +void* cs1_gemm_create_fixed(size_t n, int m) { (void)n; if(m<=0) return NULL; ++gemms; return (void*)3; } +void cs1_gemm_destroy(void* p) { (void)p; --gemms; } diff --git a/tests/qwen3_5/kernels.rs b/tests/qwen3_5/kernels.rs index 33c0ca9f..70736332 100644 --- a/tests/qwen3_5/kernels.rs +++ b/tests/qwen3_5/kernels.rs @@ -1007,78 +1007,81 @@ fn device_copy_helpers_roundtrip() { fn gdn_conv_continuation_uses_cached_history() { let st = setup(); let prefix = 64usize; - let (key_dim, value_dim, heads) = (16 * 128usize, 48 * 128usize, 48usize); - let channels = 2 * key_dim + value_dim; - let ld = channels + value_dim + 2 * heads; - let weights = to_device(&vec![bf16::from_f32(0.25); channels * 4], st); - let conv = |source: &DeviceBuffer, start: usize, rows: usize| { - let q = DeviceBuffer::new(rows * key_dim * 2).unwrap(); - let k = DeviceBuffer::new(rows * key_dim * 2).unwrap(); - let v = DeviceBuffer::new(rows * value_dim * 2).unwrap(); - // SAFETY: source contains start + rows complete pitched rows; outputs - // contain rows of the corresponding Q/K/V widths. - unsafe { - check( - (api().cs1_gdn_conv)( - source.at(start * ld * 2), - ld as i32, - weights.at(0), - q.at(0), - k.at(0), - v.at(0), - rows as i32, - key_dim as i32, - value_dim as i32, + // Decider2B uses sixteen key/value heads; preserve the 27B control too. + for (key_dim, value_dim, heads) in [(2048usize, 2048usize, 16usize), (2048, 6144, 48)] { + let channels = 2 * key_dim + value_dim; + let ld = channels + value_dim + 2 * heads; + let weights = to_device(&vec![bf16::from_f32(0.25); channels * 4], st); + let conv = |source: &DeviceBuffer, start: usize, rows: usize| { + let q = DeviceBuffer::new(rows * key_dim * 2).unwrap(); + let k = DeviceBuffer::new(rows * key_dim * 2).unwrap(); + let v = DeviceBuffer::new(rows * value_dim * 2).unwrap(); + // SAFETY: source contains start + rows complete pitched rows; outputs + // contain rows of the corresponding Q/K/V widths. + unsafe { + check( + (api().cs1_gdn_conv)( + source.at(start * ld * 2), + ld as i32, + weights.at(0), + q.at(0), + k.at(0), + v.at(0), + rows as i32, + key_dim as i32, + value_dim as i32, + st, + ), + "gdn conv continuation", + ) + .unwrap(); + } + [ + from_device(&q, rows * key_dim, st), + from_device(&k, rows * key_dim, st), + from_device(&v, rows * value_dim, st), + ] + }; + for rows in [1usize, 2, 3, 4, 65] { + let total = prefix + rows; + // Positive inputs and taps make omission of any history row observable. + let projection: Vec = random(total * ld, 37, 0.5) + .iter() + .map(|x| bf16::from_f32(x.to_f32().abs() + 0.25)) + .collect(); + let source = to_device(&projection, st); + let full = conv(&source, 0, total); + let tail = DeviceBuffer::new(3 * ld * 2).unwrap(); + let window = DeviceBuffer::new((rows + 3) * ld * 2).unwrap(); + // SAFETY: capture three prefix rows, restore them ahead of the suffix, + // and copy the suffix into the remaining non-overlapping window rows. + unsafe { + cuda::copy_dd(tail.at(0), source.at((prefix - 3) * ld * 2), 3 * ld * 2, st) + .unwrap(); + cuda::copy_dd(window.at(0), tail.at(0), 3 * ld * 2, st).unwrap(); + cuda::copy_dd( + window.at(3 * ld * 2), + source.at(prefix * ld * 2), + rows * ld * 2, st, - ), - "gdn conv continuation", - ) - .unwrap(); - } - [ - from_device(&q, rows * key_dim, st), - from_device(&k, rows * key_dim, st), - from_device(&v, rows * value_dim, st), - ] - }; - for rows in [1usize, 2, 3, 4, 65] { - let total = prefix + rows; - // Positive inputs and taps make omission of any history row observable. - let projection: Vec = random(total * ld, 37, 0.5) - .iter() - .map(|x| bf16::from_f32(x.to_f32().abs() + 0.25)) - .collect(); - let source = to_device(&projection, st); - let full = conv(&source, 0, total); - let tail = DeviceBuffer::new(3 * ld * 2).unwrap(); - let window = DeviceBuffer::new((rows + 3) * ld * 2).unwrap(); - // SAFETY: capture three prefix rows, restore them ahead of the suffix, - // and copy the suffix into the remaining non-overlapping window rows. - unsafe { - cuda::copy_dd(tail.at(0), source.at((prefix - 3) * ld * 2), 3 * ld * 2, st).unwrap(); - cuda::copy_dd(window.at(0), tail.at(0), 3 * ld * 2, st).unwrap(); - cuda::copy_dd( - window.at(3 * ld * 2), - source.at(prefix * ld * 2), - rows * ld * 2, - st, - ) - .unwrap(); - } - let restored = conv(&window, 0, rows + 3); - let missing_history = conv(&window, 3, rows); - for (i, width) in [key_dim, key_dim, value_dim].into_iter().enumerate() { - let expected = &full[i][prefix * width..]; - assert_eq!( - &restored[i][3 * width..], - expected, - "cached conv history diverges for output {i}, suffix rows={rows}" - ); - assert_ne!( - &missing_history[i][..rows.min(3) * width], - &expected[..rows.min(3) * width], - "fixture must detect the old suffix-only conv call" - ); + ) + .unwrap(); + } + let restored = conv(&window, 0, rows + 3); + let missing_history = conv(&window, 3, rows); + for (i, width) in [key_dim, key_dim, value_dim].into_iter().enumerate() { + let expected = &full[i][prefix * width..]; + assert_eq!( + &restored[i][3 * width..], + expected, + "cached conv history diverges for output {i}, suffix rows={rows}" + ); + assert_ne!( + &missing_history[i][..rows.min(3) * width], + &expected[..rows.min(3) * width], + "fixture must detect the old suffix-only conv call" + ); + } } } } @@ -1087,77 +1090,81 @@ fn gdn_conv_continuation_uses_cached_history() { #[ignore = "needs a GPU and CUA_S1_CUDA_LIB"] fn windowed_attention_matches_full_pass_bit_for_bit() { let st = setup(); - let (hq, hk, dh) = (24usize, 4usize, 256usize); - for (t, bases) in [ - (139usize, vec![64usize]), - (712, vec![64, 128]), - (972, vec![64, 896]), - (2048, vec![64, 1024]), - ] { - // gated path with a strided V buffer, exactly the model's shape - let ldv = hk * dh + 16; - let q = to_device(&random(t * hq * dh, 1, 2.0), st); - let k = to_device(&random(t * hk * dh, 2, 2.0), st); - let v = to_device(&random(t * ldv, 3, 1.0), st); - let gate = to_device(&random(t * hq * dh, 4, 3.0), st); - let full = DeviceBuffer::new(t * hq * dh * 2).unwrap(); - // SAFETY: every buffer has complete rows of the shapes above. - unsafe { - check( - (api().cs1_attention_gated)( - q.at(0), - k.at(0), - v.at(0), - ldv as i32, - gate.at(0), - full.at(0), - t as i32, - hq as i32, - hk as i32, - dh as i32, - 0.0625, - st, - ), - "full pass", - ) - .unwrap(); - } - let full_rows = from_device(&full, t * hq * dh, st); - for qb in &bases { - let win = DeviceBuffer::new(t * hq * dh * 2).unwrap(); - // SAFETY: the window reads the same buffers; rows < q_base stay unwritten. + // Canonical window indexing covers Decider2B and the existing 27B shape. + for (hq, hk, dh) in [(8usize, 2usize, 256usize), (24, 4, 256)] { + for (t, bases) in [ + (139usize, vec![64usize]), + (712, vec![64, 128]), + (972, vec![64, 896]), + (2048, vec![64, 1024]), + ] { + // gated path with a strided V buffer, exactly the model's shape + let ldv = hk * dh + 16; + let q = to_device(&random(t * hq * dh, 1, 2.0), st); + let k = to_device(&random(t * hk * dh, 2, 2.0), st); + let v = to_device(&random(t * ldv, 3, 1.0), st); + let gate = to_device(&random(t * hq * dh, 4, 3.0), st); + let full = DeviceBuffer::new(t * hq * dh * 2).unwrap(); + // SAFETY: every buffer has complete rows of the shapes above. unsafe { check( - (api().cs1_attention_gated_prefix)( + (api().cs1_attention_gated)( q.at(0), k.at(0), v.at(0), ldv as i32, gate.at(0), - win.at(0), + full.at(0), t as i32, hq as i32, hk as i32, dh as i32, 0.0625, - *qb as i32, st, ), - "windowed pass", + "full pass", ) .unwrap(); } - let win_rows = from_device(&win, t * hq * dh, st); - let mut bad = 0usize; - for row in *qb..t { - for e in 0..hq * dh { - if full_rows[row * hq * dh + e] != win_rows[row * hq * dh + e] { - bad += 1; + let full_rows = from_device(&full, t * hq * dh, st); + for qb in &bases { + let win = DeviceBuffer::new(t * hq * dh * 2).unwrap(); + // SAFETY: the window reads the same buffers; rows < q_base stay unwritten. + unsafe { + check( + (api().cs1_attention_gated_prefix)( + q.at(0), + k.at(0), + v.at(0), + ldv as i32, + gate.at(0), + win.at(0), + t as i32, + hq as i32, + hk as i32, + dh as i32, + 0.0625, + *qb as i32, + st, + ), + "windowed pass", + ) + .unwrap(); + } + let win_rows = from_device(&win, t * hq * dh, st); + let mut bad = 0usize; + for row in *qb..t { + for e in 0..hq * dh { + if full_rows[row * hq * dh + e] != win_rows[row * hq * dh + e] { + bad += 1; + } } } + assert_eq!(bad, 0, "t = {t}, q_base = {qb}: {bad} values differ"); + eprintln!( + "windowed attention t = {t}, q_base = {qb}: rows >= q_base bitwise equal" + ); } - assert_eq!(bad, 0, "t = {t}, q_base = {qb}: {bad} values differ"); - eprintln!("windowed attention t = {t}, q_base = {qb}: rows >= q_base bitwise equal"); } } } @@ -1169,6 +1176,8 @@ fn gdn_prefill_two_phase_matches_one_shot_bit_for_bit() { let d = 128usize; for (t, h, hk, qk_amp, decay_scale) in [ (107usize, 4usize, 2usize, 1.0f32, 1.0f32), + (129, 16, 16, 0.01, 1.0), + (936, 16, 16, 0.01, 1.0), (936, 4, 2, 0.01, 1.0), (3399, 3, 1, 0.01, 1.0), (65, 4, 2, 0.01, 1e-9), @@ -1325,10 +1334,15 @@ fn gdn_prefill_two_phase_matches_one_shot_bit_for_bit() { #[ignore = "needs a GPU and CUA_S1_CUDA_LIB"] fn fixed_gemm_rows_do_not_depend_on_m() { let st = setup(); - // Every projection of the 4B, 9B and 27B backbones as (N, K): the GDN input and + // Every projection of the 2B, 4B, 9B and 27B backbones as (N, K): the GDN input and // output, the attention input and output, and the MLP's gate|up and down. let shapes = [ - (12352usize, 2560usize), + (8224usize, 2048usize), + (2048, 2048), + (5120, 2048), + (12288, 2048), + (2048, 6144), + (12352, 2560), (2560, 4096), (10240, 2560), (18432, 2560), diff --git a/tests/qwen3_5/multimodal_graph.rs b/tests/qwen3_5/multimodal_graph.rs index e3979e15..7b9f7771 100644 --- a/tests/qwen3_5/multimodal_graph.rs +++ b/tests/qwen3_5/multimodal_graph.rs @@ -52,6 +52,8 @@ fn multimodal_graph_replays_current_inputs_and_clears_on_growth() -> Result<()> "capture miss must retain the eager result", ); assert_eq!(model.mm_graphs.len(), 1, "miss must cache a graph"); + assert_eq!(model.graph_stats().captures, 1); + assert_eq!(model.graph_stats().cached_shapes, 1); assert!(model.graphs.is_empty(), "text cache must remain separate"); assert!(model.graph_enabled, "capture must remain enabled"); assert_hidden_eq( @@ -60,6 +62,8 @@ fn multimodal_graph_replays_current_inputs_and_clears_on_growth() -> Result<()> "replay must equal eager exactly", ); + assert_eq!(model.graph_stats().replays, 1); + let changed_ids = [text[2], image_token, image_token, text[3]]; let changed_features: Vec = (0..2 * model.cfg.hidden) .map(|i| bf16::from_f32((i % 17) as f32 / 16.0 - 0.5)) @@ -328,7 +332,7 @@ fn graph_recording_failure_keeps_eager_result_and_disables_both_modes() -> Resul .contains("injected record failure"), "must exercise the actual recording-error path" ); - model.cache_graph(&[text.len()], multimodal, captured); + model.cache_graph(&[text.len()], multimodal, captured)?; assert_hidden_eq( &model.last_hidden(model.scratch.as_ref().unwrap(), text.len())?, &prior_eager,