teploy-gateway

The OSS AI gateway: one project key, every model, tracked — on hardware you own. Provider API keys live only in the gateway's config; projects hold scoped, revocable gateway tokens.

ANTHROPIC_API_KEY=... OPENAI_API_KEY=... teploy-gateway serve
teploy-gateway key create --project fylun

Then a project needs exactly two values:

import { createAnthropic } from "@neutron-build/ai/anthropic";
const anthropic = createAnthropic({
  baseURL: process.env.AI_GATEWAY_URL,
  apiKey: process.env.AI_GATEWAY_KEY,     // the minted project token
});
const model = anthropic("anthropic/claude-sonnet-5"); // or openai/…, groq/…

No SDK changes, either wire: the gateway serves /v1/chat/completions (OpenAI shape — also Groq, DeepSeek, xAI, anything compatible) and /v1/messages (Anthropic shape), accepts both auth header styles, rewrites provider/model to the upstream name, swaps in the real key, and streams responses through byte-for-byte.

What you get per project

• All configured models by provider/model id (bare names use the endpoint's default provider).
• Spend tracking — requests and token counts aggregated per project/provider/model (extracted from both wires, streaming included), persisted in the data dir, queryable via teploy-gateway usage --project x. A Nucleus-backed sink slots in behind the same interface.
• Failover — same-wire fallback chains ("fallbacks": {"openai/gpt-4o": ["groq/llama-3.3-70b"]}) walked on 429/5xx/unreachable; cross-wire chains are rejected at config load.
• Kill-switch — teploy-gateway key revoke <id> deadens a token instantly. Only SHA-256 hashes are stored; tokens are shown once.

Config

--config config.json (optional — built-in providers work with just env keys: ANTHROPIC_API_KEY, OPENAI_API_KEY, GROQ_API_KEY, DEEPSEEK_API_KEY, XAI_API_KEY):

{
  "providers": {
    "local": { "baseURL": "http://127.0.0.1:11434", "apiKey": "x", "wire": "openai" }
  },
  "fallbacks": { "openai/gpt-4o": ["groq/llama-3.3-70b"] },
  "max_response_bytes": 33554432,
  "body_idle_seconds": 90
}

Admin API (/admin/v1/keys, /admin/v1/usage) is guarded by a token minted 0600 to the data dir at first start. Don't bind the gateway publicly unless you put auth-terminating ingress in front of it.

The OSS gateway uses your configured provider keys. Account credits and managed billing require a separate service integration; this repository does not implement that billing contract.

Usage completeness and persistence

incompleteRequests in each usage total counts successful upstream calls whose complete token usage was unavailable, including interrupted streams and error events in an HTTP-success stream. Known token counts are retained as partial observations; missing counts are not evidence of zero cost. Chat requires both input and output counters; explicit zeros are valid. Embeddings requires input usage and treats output as an inapplicable zero. Each valid input, output or cache counter is retained independently. Complete bounded SSE usage lines in every raw provider read are retained even if the client write fails or accepts only part of that chunk. Delivery failure marks the request incomplete and stops further upstream reads. Existing historical totals cannot retrospectively identify missing usage.

For OpenAI-compatible streaming clients, send stream_options: {"include_usage": true} when your provider supports it. The gateway preserves caller options and does not inject that option into other adapters. A provider can still omit its final usage event, so inspect incompleteRequests even when requesting usage.

The database sink creates/migrates gateway_usage and creates gateway_usage_batches. A stable batch receipt and its additive usage increment commit in one transaction, making acknowledgement-loss retries idempotent across concurrent gateways. The database must implement atomic transactions, unique constraints, ON CONFLICT DO NOTHING, accurate RowsAffected, and ALTER TABLE ADD COLUMN IF NOT EXISTS. Unsupported backends that fail the startup SQL probe are refused. The probe runs receipt insert/deduplication and the actual usage UPSERT (first insert and additive conflict), checks affected rows and all counters, then rolls back and verifies neither table retains the random probe. It does not establish crash atomicity, durability, concurrent isolation or all future runtime behavior; those require acceptance on the exact backend version. Nucleus engine acceptance remains open. Receipts remain while a gateway can retry their batch and are removed after acknowledgement and local retirement. Failed cleanup or an abrupt process exit can leave orphan receipts; reclaim them only after all possible retrying gateways have drained. Pending in-memory records are not a durable outbox: abrupt process loss can still lose unflushed usage. Graceful shutdown reports final flush failures.

The proxy forwards Anthropic's anthropic-beta request header on that wire and response request IDs, Retry-After, and the listed OpenAI rate-limit headers. It excludes caller credentials, cookies and hop-by-hop metadata, including fields named by Connection.

Response limits and local storage

max_response_bytes bounds selected nonstream responses (default 32 MiB, maximum 1 GiB). An oversized response is closed, forwarded only through the ceiling, and counted incomplete without interpreting a truncated JSON prefix. body_idle_seconds bounds inactivity in incoming request reads and upstream body reads (default 90 seconds, maximum one day). Zero selects the defaults; negative values refuse startup. Progress resets the budget. Active SSE has no blanket total-response timer. Request bodies remain capped at 32 MiB; retry responses at 1 MiB. Read failures retain independently observed counts and mark accounting incomplete. Provider redirects are refused for every effective HTTP client, so configured provider credentials cannot follow a redirect.

SSE accounting retains at most 128 KiB per line. A line exceeding that ceiling is discarded through newline or EOF; its prefix is never parsed as a complete observation. Malformed or oversized usage/error candidates make accounting incomplete for the stream, preserving earlier valid counts even after later valid events. Bounded member/type state survives arbitrary JSON spaces, tabs and carriage returns across raw reads, without matching ordinary content strings. Ordinary comments, content deltas and control lines do not invalidate usage. Successfully delivered response bytes remain unchanged.

The daemon holds an exclusive process lease on the data directory before opening keys, usage or the admin-token file, in either sink mode. A second process using that directory refuses startup. Supported local-file operation requires Linux or macOS, a local filesystem with working flock, atomic rename and file/directory synchronization, and directory durability. Newly created data-directory components and their first existing ancestor are synchronized before stores open. Network/shared filesystems and independent file-store writers are unsupported. Library callers must hold AcquireDataDirLease across all store lifetimes. The persistent .gateway.lock inode must not be removed while a daemon runs; closing or crashing the owning process releases the kernel lease.

Usage, keys and newly minted admin tokens use unique private temporary files, checked full writes, file sync/close, atomic rename and parent-directory sync. A definite failure before rename leaves published keys unchanged. A failure after rename is an uncertain durability outcome: the operation returns an error, but memory retains the visible key mutation, including revocation. Mint/rotate do not return a new plaintext token on that error; inspect the key list before retrying and revoke an inaccessible replacement if necessary. Usage retries its cumulative snapshot rather than incrementing it again. Existing invalid or unreadable admin-token files refuse startup instead of replacing credentials. These mechanisms still require separate restart/power-loss acceptance on the supported filesystem; buffered unflushed usage remains memory-only.

Liveness, readiness and shutdown

GET /health preserves HTTP 200 / status: ok as liveness and includes an accounting state. GET /readyz returns 503 while accounting has a retained flush failure or retryable pending database batches, the sink is closed, or the database Ping fails. A successful Ping alone cannot clear a failed flush; a successful retry clears that failure. Normal buffered records awaiting the next flush are expected and do not degrade readiness. The Teploy template and container health probe use /readyz for deployment readiness; use /health when a separate process-liveness probe is needed. Health payloads omit backend error text and credentials. Admin database query failures still return 503.

Shutdown drains for up to 30 seconds, then closes connections and allows at most two additional seconds for handlers to settle before the final sink close. An expired settle ceiling reports outstanding handlers and returns an error; late records may be refused by the closed sink. Cooperative completion and final flush errors are preserved. Startup SQL-probe failures retain their random probe ID and any cleanup failure; exact residue queries are diagnostic only and run only after a safe terminal transaction outcome. No arbitrary backend rows are deleted on rejection. Real disconnect/backpressure, engine failure, image execution and power-loss acceptance remain independent gates.