-Each instance writes updates to redis -
- - -### Stage 2. A single instance flushes the redis queue to the DB - -A single instance will acquire a lock on the DB and flush all elements in the redis queue to the DB. - -- 1 instance will attempt to acquire the lock for the DB update job -- The status of the lock is stored in redis -- If the instance acquires the lock to write to DB - - It will read all updates from redis - - Aggregate all updates into 1 transaction - - Write updates to DB - - Release the lock -- Note: Only 1 instance can acquire the lock at a time, this limits the number of instances that can write to the DB at once - - --A single instance flushes the redis queue to the DB -
- - -## Usage - -### Required components - -- Redis -- Postgres - -### Setup on LiteLLM config - -You can enable using the redis buffer by setting `use_redis_transaction_buffer: true` in the `general_settings` section of your `proxy_config.yaml` file. - -Note: This setup requires litellm to be connected to a redis instance. - -```yaml showLineNumbers title="litellm proxy_config.yaml" -general_settings: - use_redis_transaction_buffer: true - -litellm_settings: - cache: True - cache_params: - type: redis - supported_call_types: [] # Optional: Set cache for proxy, but not on the actual llm api call -``` - -## Monitoring - -LiteLLM emits the following prometheus metrics to monitor the health/status of the in memory buffer and redis buffer. - - -| Metric Name | Description | Storage Type | -|-----------------------------------------------------|-----------------------------------------------------------------------------|--------------| -| `litellm_pod_lock_manager_size` | Indicates which pod has the lock to write updates to the database. | Redis | -| `litellm_in_memory_daily_spend_update_queue_size` | Number of items in the in-memory daily spend update queue. These are the aggregate spend logs for each user. | In-Memory | -| `litellm_redis_daily_spend_update_queue_size` | Number of items in the Redis daily spend update queue. These are the aggregate spend logs for each user. | Redis | -| `litellm_in_memory_spend_update_queue_size` | In-memory aggregate spend values for keys, users, teams, team members, etc.| In-Memory | -| `litellm_redis_spend_update_queue_size` | Redis aggregate spend values for keys, users, teams, etc. | Redis | - - -## Troubleshooting: Redis Connection Errors - -You may see errors like: - -``` -LiteLLM Redis Caching: async async_increment() - Got exception from REDIS No connection available., Writing value=21 -LiteLLM Redis Caching: async set_cache_pipeline() - Got exception from REDIS No connection available., Writing value=None -``` - -This means all available Redis connections are in use, and LiteLLM cannot obtain a new connection from the pool. This can happen under high load or with many concurrent proxy requests. - -**Solution:** - -- Increase the `max_connections` parameter in your Redis config section in `proxy_config.yaml` to allow more simultaneous connections. For example: - -```yaml -litellm_settings: - cache: True - cache_params: - type: redis - max_connections: 100 # Increase as needed for your traffic -``` - -Adjust this value based on your expected concurrency and Redis server capacity. - diff --git a/docs/proxy/db_info.md b/docs/proxy/db_info.md index 5ef9fa550..46d1ffbfa 100644 --- a/docs/proxy/db_info.md +++ b/docs/proxy/db_info.md @@ -47,27 +47,12 @@ You can see the full DB Schema [here](https://github.com/BerriAI/litellm/blob/ma | Table Name | Description | Row Insert Frequency | |------------|-------------|---------------------| | LiteLLM_SpendLogs | Detailed logs of all API requests. Records token usage, spend, and timing information. Tracks which models and keys were used. | **Medium - this is a batch process that runs on an interval.** | +| LiteLLM_DailyUserSpend and siblings (DailyTeamSpend, DailyOrgSpend, DailyTagSpend, DailyEndUserSpend, DailyAgentSpend) | Pre-aggregated daily spend rollups per user, team, org, tag, end user, and agent; the Admin UI Usage views read these aggregates rather than scanning SpendLogs. | Low - one row per entity per day, updated in batches | | LiteLLM_AuditLog | Tracks changes to system configuration. Records who made changes and what was modified. Maintains history of updates to teams, users, and models. | **Off by default**, **High - Runs on every change to an entity** | ## Disable `LiteLLM_SpendLogs` -You can disable spend_logs and error_logs by setting `disable_spend_logs` and `disable_error_logs` to `True` on the `general_settings` section of your proxy_config.yaml file. - -```yaml -general_settings: - disable_spend_logs: True # Disable writing spend logs to DB - disable_error_logs: True # Only disable writing error logs to DB, regular spend logs will still be written unless `disable_spend_logs: True` -``` - -### What is the impact of disabling these logs? - -When disabling spend logs (`disable_spend_logs: True`): -- You **will not** be able to view Usage on the LiteLLM UI -- You **will** continue seeing cost metrics on s3, Prometheus, Langfuse (any other Logging integration you are using) - -When disabling error logs (`disable_error_logs: True`): -- You **will not** be able to view Errors on the LiteLLM UI -- You **will** continue seeing error logs in your application logs and any other logging integrations you are using +Set `disable_spend_logs: True` or `disable_error_logs: True` under `general_settings` to stop writing those tables. With spend logs disabled you lose per-request log detail in the UI but keep cost metrics in your logging integrations (s3, Prometheus, Langfuse); with error logs disabled you lose the Errors view in the UI but keep errors in application logs and other logging integrations. See [keeping error logs out of the database](./prod.md#keep-error-logs-out-of-the-database) in the production checklist. ## Migrating Databases diff --git a/docs/proxy/db_read_replica.md b/docs/proxy/db_read_replica.md index 9a258afef..6a9ce66b7 100644 --- a/docs/proxy/db_read_replica.md +++ b/docs/proxy/db_read_replica.md @@ -58,11 +58,11 @@ worst it degrades to single-database performance. ## RDS IAM authentication When `IAM_TOKEN_DB_AUTH=True`, both the writer and the reader refresh their -IAM tokens independently on the same ~12-minute cadence. The reader does not -need parallel `DATABASE_HOST_READ_REPLICA` / `DATABASE_USER_READ_REPLICA` -env vars β host, port, user, and database name are parsed once from -`DATABASE_URL_READ_REPLICA` at startup, and only the IAM token rotates after -that. +IAM tokens independently on the same ~12-minute cadence. You can supply the +reader either as a single `DATABASE_URL_READ_REPLICA` or assembled from the +component vars (`DATABASE_HOST_READ_REPLICA`, `DATABASE_USER_READ_REPLICA`, +and friends); the token-refresh path re-parses the resulting reader URL, +so only the IAM token rotates after startup. This pairs naturally with Aurora's reader endpoint, which resolves to the reader instances in the cluster. diff --git a/docs/proxy/deploy.md b/docs/proxy/deploy.md index d0634a563..be5a5a4af 100644 --- a/docs/proxy/deploy.md +++ b/docs/proxy/deploy.md @@ -1,189 +1,204 @@ +--- +title: Production Deployment +description: Production deployment guide for LiteLLM on AWS, GCP, Azure, or any Kubernetes cluster, with Helm charts and official Terraform modules. +--- + import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import Image from '@theme/IdealImage'; +import { CloudArchitectureSelector } from '@site/src/components/CloudArchitecture'; -# Docker, Helm, Terraform - -:::info No Limits on LiteLLM OSS -There are **no limits** on the number of users, keys, or teams you can create on LiteLLM OSS. -::: +# Production Deployment -You can find the Dockerfile to build litellm proxy [here](https://github.com/BerriAI/litellm/blob/main/Dockerfile) +Production deployment guide for AWS, Google Cloud, Azure, or any Kubernetes cluster. For a first deployment on a single machine, start with the [Quickstart](./docker_quick_start.md); this page picks up where it ends. -Official images are published to `ghcr.io/berriai` (`litellm`, `litellm-database` which bundles prisma for use with Postgres, and `litellm-non_root`) and mirrored at `docker.litellm.ai/berriai`. The snippets below use the `docker.litellm.ai` mirror. +There are two supported paths. If you run Kubernetes, [deploy with Helm](#deploy-with-helm) on EKS, GKE, or AKS; the install is the same on every cloud, only the data stores and ingress differ. If you do not run Kubernetes, AWS and GCP have [official Terraform modules](#deploy-with-terraform-aws-and-gcp) that stand up the entire stack; Azure has no Terraform module, so AKS with Helm is the supported path there. -> Note: for production sizing, see [machine specifications](./prod.md#2-recommended-machine-specifications). +## Architecture -## Quick Start - -:::info -Facing issues with pulling the docker image? Email us at support@berri.ai. -::: - -
+
-### Helm Chart
+To connect the database, switch the image to `docker.litellm.ai/berriai/litellm-database` and add `DATABASE_URL` and `LITELLM_MASTER_KEY` to the Secret; nothing else in the manifest changes.
-:::info
+## Deploy with Terraform (AWS and GCP)
-[BETA] Helm Chart is BETA. If you run into an issues/have feedback please let us know [https://github.com/BerriAI/litellm/issues](https://github.com/BerriAI/litellm/issues)
+The official modules deploy the full microservices stack (network, database, Redis, object storage, secrets, compute, load balancer, and a migrations job that runs before the services start) and are published to the Terraform Registry:
-:::
-
-The canonical chart lives at [`deploy/charts/litellm-helm`](https://github.com/BerriAI/litellm/tree/main/deploy/charts/litellm-helm) in the litellm repo and is published as an OCI artifact at `oci://ghcr.io/berriai/litellm-helm`, the published chart versions carry litellm release numbers (for example, `1.90.2`). This section covers a local quickstart; for a production install on EKS, GKE, or AKS with managed Postgres and Redis, see [Deploy to Cloud](./deploy_cloud.md#deploy-with-helm).
+- [`BerriAI/litellm/aws`](https://registry.terraform.io/modules/BerriAI/litellm/aws/latest)
+- [`BerriAI/litellm/google`](https://registry.terraform.io/modules/BerriAI/litellm/google/latest)
Full manifest (ConfigMap, Secret, Deployment, Service)
```yaml apiVersion: v1 @@ -192,12 +207,11 @@ metadata: name: litellm-config-file data: config.yaml: | - model_list: + model_list: - model_name: gpt-4o litellm_params: - model: azure/gpt-4o-ca - api_base: https://my-endpoint-canada-berri992.openai.azure.com/ - api_key: os.environ/CA_AZURE_OPENAI_API_KEY + model: openai/gpt-4o + api_key: os.environ/OPENAI_API_KEY --- apiVersion: v1 kind: Secret @@ -205,7 +219,7 @@ type: Opaque metadata: name: litellm-secrets data: - CA_AZURE_OPENAI_API_KEY: bWVvd19pbV9hX2NhdA== # your api key in base64 + OPENAI_API_KEY: bWVvd19pbV9hX2NhdA== # your api key in base64 --- apiVersion: apps/v1 kind: Deployment @@ -214,7 +228,7 @@ metadata: labels: app: litellm spec: - replicas: 1 + replicas: 2 selector: matchLabels: app: litellm @@ -269,502 +283,124 @@ spec: type: NodePort ``` -Port-forward to reach the proxy locally: - -```bash -kubectl port-forward service/litellm-service 4000:4000 -``` - -To connect a database (Virtual Keys, spend tracking), use the `docker.litellm.ai/berriai/litellm-database` image and add `DATABASE_URL` and `LITELLM_MASTER_KEY` to the Secret; nothing else in the manifest changes. See [Deploy with Database](#deploy-with-database). To run more than one instance behind a load balancer, see [Multi-region and scaling](./multi_region.md). - -:::info -To avoid issues with predictability, difficulties in rollback, and inconsistent environments, pin a version or SHA digest (for example, `litellm:main-v1.90.2` or `litellm@sha256:12345abcdef...`) instead of `litellm:latest`. -::: - +
+ 
+ 
+
-```shell
-$ uv tool install 'litellm[proxy]'
-```
-
-:::warning Requires Python 3.10+
-LiteLLM 1.84.0 and newer require Python 3.10 or higher (`requires-python >=3.10`). `uv tool install` and the `install.sh` script above handle this for you by provisioning a compatible Python automatically. A bare `pip install 'litellm[proxy]'` does not; on Python 3.9 pip silently resolves down to the last release that still allowed 3.9, which is 1.83.9, with no error. If you pinned to an old version unexpectedly, check `python --version` and upgrade to 3.10+ (or use uv), then reinstall
-:::
+For the rest of this guide, use your deployment's URL wherever you see `http://localhost:4000`.
+
+
+
+
**Option 2: Without UI (API-only deployment)**
If you don't need the admin UI, you can run with minimal configuration:
@@ -435,7 +365,7 @@ securityContext:
The proxy will log a warning about the UI but API endpoints will work normally.
-#### Environment Variables for Read-Only Filesystems
+Environment variables for read-only filesystems:
| Variable | Purpose | Default |
|----------|---------|---------|
@@ -445,23 +375,38 @@ The proxy will log a warning about the UI but API endpoints will work normally.
| `PRISMA_BINARY_CACHE_DIR` | Prisma binary cache | System default |
| `XDG_CACHE_HOME` | General cache directory | System default |
-#### Important Notes
+Notes: always set `LITELLM_MIGRATION_DIR` to a writable emptyDir path, and set `PRISMA_BINARY_CACHE_DIR` and `XDG_CACHE_HOME` to writable paths. If using a custom `server_root_path`, you must pre-process UI files in your Dockerfile as the proxy cannot modify files at runtime with a read-only filesystem. The UI is automatically detected as pre-restructured if it contains a `.litellm_ui_ready` marker file (created by the official Docker images).
-1. **Migrations**: Always set `LITELLM_MIGRATION_DIR` to a writable emptyDir path
-2. **Prisma Cache**: Set `PRISMA_BINARY_CACHE_DIR` and `XDG_CACHE_HOME` to writable paths
-3. **Server Root Path**: If using a custom `server_root_path`, you must pre-process UI files in your Dockerfile as the proxy cannot modify files at runtime with read-only filesystem
-4. **Automatic Detection**: The UI is automatically detected as pre-restructured if it contains a `.litellm_ui_ready` marker file (created by the official Docker images)
+## Verify production readiness
-## Extras
-### Expected Performance in Production
+### Expected performance
-See benchmarks [here](../benchmarks#performance-metrics)
+See benchmarks [here](../benchmarks#performance-metrics).
-### Verifying Debugging logs are off
+### Confirm debug logging is off
+
+You should only see the following level of details in logs on the proxy server:
-You should only see the following level of details in logs on the proxy server
```shell
# INFO: 192.168.2.205:11774 - "POST /chat/completions HTTP/1.1" 200 OK
# INFO: 192.168.2.205:34717 - "POST /chat/completions HTTP/1.1" 200 OK
# INFO: 192.168.2.205:29734 - "POST /chat/completions HTTP/1.1" 200 OK
```
+
+## Deployment FAQ
+
+**Q: Is Postgres the only supported database, or do you support other ones (like Mongo)?**
+
+A: We explored MySQL but that was hard to maintain and led to bugs for customers. Currently, PostgreSQL is our primary supported database for production deployments.
+
+Because LiteLLM talks to the database through Prisma over the PostgreSQL wire protocol, any Postgres-wire-compatible distributed SQL database works as a drop-in replacement. [YugabyteDB](https://www.yugabyte.com/) is used in production this way; point `DATABASE_URL` at its YSQL endpoint (`postgresql://Full Deployment manifest (initContainer, env, securityContext, volumes)
+ ```yaml apiVersion: apps/v1 kind: Deployment @@ -419,6 +347,8 @@ spec: sizeLimit: 64Mi ``` +}){kind=icon}
+ );
+}
+
+function Clients({ label = 'Clients (OpenAI SDK, LangChain, curl)' }: { label?: string }) {
+ return (
+
+
+ );
+}
+
+function Node({
+ icon,
+ title,
+ subtitle,
+ accent,
+ small,
+}: {
+ icon: string;
+ title: string;
+ subtitle?: string;
+ accent?: 'blue' | 'green';
+ small?: boolean;
+}) {
+ const cls = [
+ styles.node,
+ accent === 'blue' ? styles.nodeAccent : '',
+ accent === 'green' ? styles.nodeGreen : '',
+ small ? styles.nodeSmall : '',
+ ].join(' ');
+ return (
+
+
+
+ {label}
+
+
+ );
+}
+
+function StepNode({
+ n,
+ title,
+ subtitle,
+ accent,
+}: {
+ n?: number;
+ title: string;
+ subtitle?: string;
+ accent?: 'blue' | 'green';
+}) {
+ const cls = [
+ styles.node,
+ styles.nodeSmall,
+ accent === 'blue' ? styles.nodeAccent : '',
+ accent === 'green' ? styles.nodeGreen : '',
+ ].join(' ');
+ return (
+
+ {title}
+ {subtitle && {subtitle}}
+
+
+ {n !== undefined && {n}}
+
+ );
+}
+
+function PillsNode({
+ title,
+ subtitle,
+ pills,
+}: {
+ title: string;
+ subtitle?: string;
+ pills: string[];
+}) {
+ return (
+
+ {title}
+ {subtitle && {subtitle}}
+
+
+
+ );
+}
+
+function ConnectorDown({ label }: { label?: string }) {
+ if (!label) return ;
+ return (
+
+
+
+ {title}
+ {subtitle && {subtitle}}
+
+
+ {pills.map((p) => (
+
+ {p}
+
+ ))}
+
+
+ {label}
+
+
+ );
+}
+
+function Branch({ legs }: { legs: number }) {
+ const positions =
+ legs === 2 ? ['30%', '70%'] : legs === 3 ? ['20%', '50%', '80%'] : ['50%'];
+ return (
+
+
+
+ {positions.map((p) => (
+
+ ))}
+
+ );
+}
+
+function Box({
+ name,
+ badge,
+ badgeColor,
+ children,
+}: {
+ name: string;
+ badge?: string;
+ badgeColor?: 'blue' | 'green' | 'orange';
+ children: React.ReactNode;
+}) {
+ const badgeCls =
+ badgeColor === 'green'
+ ? styles.badgeGreen
+ : badgeColor === 'orange'
+ ? styles.badgeOrange
+ : styles.badgeBlue;
+ return (
+
+
+ );
+}
+
+/* ββββββββββββββββββββββ Life of a Request ββββββββββββββββββββββ */
+
+export function RequestFlowDiagram() {
+ return (
+
+ {name}
+ {badge && {badge}}
+
+ {children}
+
+
+ );
+}
+
+/* ββββββββββββββββββββββ Router: fallbacks and retries ββββββββββββββββββββββ */
+
+export function RouterFlowDiagram() {
+ return (
+
+
+
+
+
+
+
+
+ After the response is returned to the client, spend logging, rate-limit accounting, and
+ logging callbacks all run as asynchronous background tasks; no database write sits in
+ the request path.
+
+
+
+
+ );
+}
+
+/* ββββββββββββββββββββββ Image URL handling ββββββββββββββββββββββ */
+
+export function ImageFlowDiagram() {
+ return (
+
+
+
+
+ Retries stay inside the model group that failed; fallbacks leave it for the next group
+ in your fallbacks configuration.
+
+
+
+
+ );
+}
+
+/* ββββββββββββββββββββββ Tenancy hierarchy ββββββββββββββββββββββ */
+
+export function TenancyDiagram() {
+ return (
+
+
+
+
+
+
+
+
+
+
+ );
+}
diff --git a/src/components/CloudArchitecture/index.tsx b/src/components/CloudArchitecture/index.tsx
index 50c2c2dd1..76a4446ca 100644
--- a/src/components/CloudArchitecture/index.tsx
+++ b/src/components/CloudArchitecture/index.tsx
@@ -3,3 +3,9 @@ export {
CloudArchitectureSelector,
MultiRegionArchitecture,
} from './CloudArchitecture';
+export {
+ RequestFlowDiagram,
+ RouterFlowDiagram,
+ ImageFlowDiagram,
+ TenancyDiagram,
+} from './ConceptDiagrams';
diff --git a/src/components/CloudArchitecture/styles.module.css b/src/components/CloudArchitecture/styles.module.css
index 857672bba..f2b7c2a93 100644
--- a/src/components/CloudArchitecture/styles.module.css
+++ b/src/components/CloudArchitecture/styles.module.css
@@ -500,3 +500,17 @@
gap: 0.6rem;
}
}
+
+.stepChip {
+ width: 26px;
+ height: 26px;
+ border-radius: 50%;
+ background: var(--ca-accent);
+ color: #fff;
+ display: flex;
+ align-items: center;
+ justify-content: center;
+ font-size: 0.75rem;
+ font-weight: 700;
+ flex-shrink: 0;
+}
diff --git a/static/docker-compose.yml b/static/docker-compose.yml
new file mode 100644
index 000000000..e87018e1e
--- /dev/null
+++ b/static/docker-compose.yml
@@ -0,0 +1,39 @@
+# LiteLLM quickstart stack: the gateway plus a Postgres database that stores
+# models, virtual keys, and spend logs. Used by https://docs.litellm.ai/docs/proxy/docker_quick_start
+#
+# Start it directly:
+# curl -sSL https://docs.litellm.ai/docker-compose.yml | docker compose -f - up -d
+#
+# The credentials below are placeholders for local evaluation. Before any real
+# use, download this file and set LITELLM_MASTER_KEY and LITELLM_SALT_KEY to
+# long random values, and pin the image to a release tag instead of `latest`.
+services:
+ litellm:
+ image: docker.litellm.ai/berriai/litellm-database:latest
+ ports:
+ - "4000:4000"
+ environment:
+ LITELLM_MASTER_KEY: sk-1234
+ LITELLM_SALT_KEY: sk-XXXXXXXXXXXXXXXX
+ DATABASE_URL: postgresql://litellm:litellm@db:5432/litellm
+ STORE_MODEL_IN_DB: "True"
+ depends_on:
+ db:
+ condition: service_healthy
+
+ db:
+ image: postgres:16
+ environment:
+ POSTGRES_USER: litellm
+ POSTGRES_PASSWORD: litellm
+ POSTGRES_DB: litellm
+ healthcheck:
+ test: ["CMD-SHELL", "pg_isready -U litellm"]
+ interval: 5s
+ timeout: 5s
+ retries: 10
+ volumes:
+ - postgres_data:/var/lib/postgresql/data
+
+volumes:
+ postgres_data:
+
+
+
+
+
+
+
+
+
+
+
+ Every request's spend is attributed to its key, user, team, and organization at once,
+ and budgets are enforced at each level; a request is blocked when any level on its path
+ is over budget. Teams are the top-level boundary in open source; Organizations add the
+ outer layer and are an enterprise feature.
+
+
+