Skip to content

feat(openai): add LLM.with_cloudflare() for Cloudflare AI Gateway#6131

Open
mcdgavin wants to merge 1 commit into
livekit:mainfrom
mcdgavin:feat/openai-with-cloudflare
Open

feat(openai): add LLM.with_cloudflare() for Cloudflare AI Gateway#6131
mcdgavin wants to merge 1 commit into
livekit:mainfrom
mcdgavin:feat/openai-with-cloudflare

Conversation

@mcdgavin

@mcdgavin mcdgavin commented Jun 16, 2026

Copy link
Copy Markdown

Summary

Closes #6101. Adds openai.LLM.with_cloudflare(...), a convenience constructor that routes
LLM traffic through the Cloudflare AI Gateway
unified OpenAI-compatible endpoint — mirroring the existing with_openrouter / with_azure
helpers.

from livekit.plugins import openai

# Minimal: account_id from CLOUDFLARE_ACCOUNT_ID, gateway token from CLOUDFLARE_AI_GATEWAY_TOKEN
llm = openai.LLM.with_cloudflare(model="openai/gpt-4o")

# Explicit, with bring-your-own provider key and per-request gateway options
llm = openai.LLM.with_cloudflare(
    model="workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast",
    account_id="<account_id>",
    gateway_id="default",
    cf_aig_token="<gateway_token>",
    api_key="<provider_key>",
    gateway_options={"cache_ttl": 3600, "max_attempts": 3, "backoff": "exponential"},
)

Design notes

  • Dual auth. Cloudflare supports two layers: cf-aig-authorization (gateway token) and
    Authorization (BYO downstream provider key); at least one is required. In gateway-stored-keys
    mode the OpenAI SDK still needs a non-empty Authorization, so a "cloudflare" placeholder is
    used — the same pattern as with_ollama.
  • Per-request options are headers, not body. Unlike OpenRouter (JSON body fields),
    Cloudflare's controls are cf-aig-* HTTP headers. gateway_options is a
    CloudflareGatewayOptions TypedDict covering caching, retries, timeout, metadata, and custom
    cost; set fields map to the corresponding headers (metadata/custom_cost JSON-encoded). A
    grouped TypedDict (over loose kwargs) keeps the signature to one extra param while typing the
    structured values. Validation is type-level only; numeric ranges pass through to Cloudflare.
  • Convenience. The gateway URL is built from account_id (env CLOUDFLARE_ACCOUNT_ID) and
    gateway_id (default "default"), with a base_url override.

Testing

tests/test_openai_with_cloudflare.py (hermetic, --unit): URL building from
account_id/gateway_id, CLOUDFLARE_ACCOUNT_ID env fallback, base_url override, BYOK vs
gateway-stored-keys auth, the full gateway_options → cf-aig-* header mapping (incl. JSON
encoding and the skip_cache-only-when-true rule), and all ValueError paths. make check
passes (ruff format + lint, strict mypy).

Out of scope

STT/TTS via Cloudflare Workers AI (#3163), tracked separately.

Adds a with_cloudflare() convenience constructor that routes LLM traffic through the
Cloudflare AI Gateway unified OpenAI-compatible endpoint, mirroring with_openrouter /
with_azure.

- Builds the gateway URL from account_id (env CLOUDFLARE_ACCOUNT_ID) and gateway_id
  (default "default"), with a base_url override, mirroring with_azure.
- Dual auth: cf_aig_token -> cf-aig-authorization, api_key -> Authorization (BYOK);
  requires at least one. Placeholder key for gateway-stored-keys mode (with_ollama precedent).
- gateway_options (CloudflareGatewayOptions TypedDict) maps caching/retry/timeout/metadata/
  custom-cost to cf-aig-* request headers.
- Lists Cloudflare in the plugin's provider docstring; hermetic unit tests for URL building,
  auth modes, and the gateway-options header mapping.

Closes livekit#6101

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

@CLAassistant

CLAassistant commented Jun 16, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cloudeflare AI gateway intregation like openrouter

2 participants