Oh My Pi is the coding agent. NInfer keeps your session's continuation state resident on your GPU. OMP NInfer is the qualified release that binds them with one hash-pinned Qwen3.8 27B model on an RTX 5090, 4090, or 3090 you own.
Every install route in the guide was run end to end on v0.12.2 with unmodified Oh My Pi 18.12.1 — 31 steps, three client platforms, both native lanes (receipt).
Download v0.12.2 · Lanes · Facts · Compare · Benchmarks · Architecture · Security · Troubleshooting · Roadmap · Changelog
A long coding session is 100K tokens of transcript, thinking, tool calls, and images. Sent to a cloud provider, every token is metered and every file leaves your machine. Sent to a stateless local server, every turn re-sends the whole transcript. A live NInfer, llama.cpp, or vLLM process already avoids recomputing an unchanged prefix while it stays up; that is prior art, and this project does not claim it. What OMP NInfer adds is continuation you can rely on:
- Explicit and durable. Oh My Pi drives NInfer through stateful OpenAI Responses
(
previous_response_id); forks and rollback are qualified, and checkpoints restore after a server restart. Oh My Pi commits the transcript first, so losing retained state degrades to a replay, never a broken session. - Private and fail-closed. Both endpoints bind loopback only, every request is bearer-authenticated, and model fallback is off: an unreachable GPU fails the turn instead of answering from a cloud model. This is acceptance-tested, not configured.
- Exact and verifiable. Model SHA-256, container digest and SBOM, native package and client
checksums, and every qualification receipt are bound in one manifest;
python3 scripts/verify_release.py --require-readyproves your clone is the qualified release.
Recorded on v0.3.0 (RTX 5090): bug fixed, tests green, stateful follow-up. 15 s, idle gaps compressed, no restart shown · MP4 · provenance
Use OMP NInfer when you run Oh My Pi, own an RTX 5090, 4090, or 3090 on Windows 11, want Qwen3.8 27B, and care about long private sessions that survive a server restart.
Use something else when you want a broad model catalog (Ollama), a desktop GUI (LM Studio), the widest hardware reach or Apple silicon (llama.cpp), or multi-user throughput (vLLM). Candid comparison: decision guide.
| Lane | Host and client | What you get | Measured |
|---|---|---|---|
| RTX 5090 | Windows 11 + Docker Desktop/WSL2 container; Windows client local, macOS over a manual SSH tunnel (preview), Linux client under WSL2 | 131,072-token context, two requests in flight, DFlash2 draft window 7, vision + tools + preserved thinking | 179.79 tok/s decode; exact 130,048-token retrieval in 58.89 s cold (receipt) |
| RTX 4090 | Native Windows 11 package; Windows client local | 131,072-token context, one request at a time, MTP3, text + tools | 158.66 tok/s C1 decode; exact 130,048-token retrieval in 91.03 s (receipt) |
| RTX 3090 | Native Windows 11 package; Windows client local | 131,072-token context, one request at a time, MTP3, 300 W serving cap, text + tools | 102.41 tok/s C1 decode; exact 130,048-token retrieval in 220.52 s (receipt) |
Both native lanes install on one two-GPU host, each bound to its GPU UUID, and run one at a time on port 18082; the RTX 5090 lane serves its own 20.4 GB model artifact and the native lanes share an 18.2 GB one. Checkpoints are best effort and restore only on the same runtime, model, profile, and credentials. Exact component, source, and model identities: facts · manifest.
- Choose your lane and check its prerequisites.
- Clone tag
v0.12.2and runpython3 scripts/verify_release.py --require-ready. - Install the stock Oh My Pi 18.12.1 client with the fail-closed config and per-model fragment.
- Run the acceptance checks at the end of your lane's route: tool turn, continuation, restart survival, fail closed.
Upgrading from v0.12.1 or v0.12.0 changes only the client; 18.12.1 fixes the native Windows
client's C: path resolution and two omp -p false-success exits. Native owners staying on
v0.10.0 keep the v0.10.0 guide.
Oh My Pi (coding-agent client) → OMP NInfer (qualified appliance and release layer) → NInfer (inference engine) → Qwen3.8 27B (served model).
- Oh My Pi owns the truth. Transcript, tools, branches, and replay. A turn advances provider state only after a complete stream and a durable transcript write.
- NInfer owns inference and retained state. The C++/CUDA engine from Neroued/ninfer and its GPU ports, plus this project's explicit Responses state and checkpoints. Retained state accelerates; it is never the source of truth.
- The manifest owns identity.
releases/v0.12.2/manifest.jsonbinds client, runtime, model, profile, and receipts;readyrequires composed external acceptance.
Architecture · Security model · Release lifecycle
Each lane is measured on its own gate, machine, and profile; the strip is not a ranking. Method, per-lane charts, the warm-versus-cold boundary rules, model quality, and the community table: benchmarks.
python3 scripts/verify_release.py --require-ready
python3 -m unittest discover -s tests -vBoth pass on a tagged release. A ready manifest binds the upstream Oh My Pi binaries,
compatibility matrix, container digest and SBOM or native package checksums, lane-specific model
SHA-256, per-lane qualification receipts, and owner-operated acceptance of every documented route.
Lifecycle and cut rules: releases.
| Document | What it answers |
|---|---|
| Quickstart | Which lane, prerequisites, exact install, acceptance checks |
| Facts | Current identities, limits, and non-claims in one place |
| Decision guide | This versus Ollama, LM Studio, llama.cpp, vLLM |
| Compatibility | Qualified client and runtime combinations (generated) |
| Architecture | State ownership, loopback topology, identity chain |
| Benchmarks | Receipts, method, charts, model quality, community results |
| Performance | Kernel and schedule experiment ledger (EXP-001..EXP-095) |
| Security | Threat model and boundaries |
| Troubleshooting | Failure signatures and recovery |
| Related work | Prior art and the NInfer port family |
| Upstream | Upstream watch and fork points |
| Releases | Version history and how a release is cut |
| Roadmap | Remaining work and explicit non-promises |
| Changelog | What changed, per release |
| Media provenance | Checksums and capture notes for every recording |
- Ran a clean install? Report time-to-first-turn and every manual step; install friction is a bug.
- Measured your card? Submit the performance result form after the acceptance checks pass.
- Work on CUDA kernels? Pick a bottleneck from the performance program and send before/after receipts.
- Need another model or profile? File a model/profile request.
Routing rules: CONTRIBUTING.md. Strip keys, hostnames, prompts, outputs, and raw
logs before filing. Support covers v0.12.2 and its exact profiles for a single local owner; no SLA.
Vulnerabilities: private reporting, never a public issue.
- Oh My Pi by can1357, built on Pi by Mario Zechner (MIT). The coding agent this appliance serves; since v0.8.0 the client is the unmodified upstream binary and this project publishes no client.
- NInfer by Neroued (Apache-2.0). The C++/CUDA engine,
.ninferformat, MTP and DFlash2 speculative decoding, and evaluation campaigns. Public runtime source for all three lanes: alphastorm/ninfer. - The Qwen team (Apache-2.0). Qwen3.8 27B, served from the registered conversions at neroued/Qwen3.8-27B-NInfer.
- UDPSendToFailed/ninfer-4090 and Don-Chad/ninfer-3090 (Apache-2.0). The RTX 4090 and RTX 3090 ports the native lanes build on.
- Gated DeltaNet (arXiv: 2412.06464), Tri Dao's ReplaySSM note,
Z-Lab's DFlash, Unsloth's NVFP4 weights, and vendored
utf8proc,nlohmann/json, andcpp-httplib; tracked in upstream.
Community project; not affiliated with or endorsed by Oh My Pi, Qwen, or NVIDIA.
MIT. NInfer and the Qwen artifacts retain their own licenses and notices.