New Feature: Add optional LiteLLM Docker Compose setup for local LLM gateway#526
New Feature: Add optional LiteLLM Docker Compose setup for local LLM gateway#526marazik wants to merge 2 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new Docker-based deployment configuration for DeepWiki using LiteLLM and Ollama, including a multi-stage Dockerfile, a Docker Compose setup, and LiteLLM configuration files. The review identifies several security risks involving hardcoded API keys and database passwords that should be moved to environment variables. Additionally, there are networking issues in the LiteLLM configuration regarding the Ollama API base, and the Dockerfile is missing several optimizations and dependencies found in the original version, such as build caching, memory limits, and CA certificates. Feedback also suggests avoiding pulling large LLM models during the build process to improve image portability and build reliability.
| # LiteLLM on host OS http://host.docker.internal:4000 | ||
| # LiteLLM on another server http://server-ip:4000 | ||
| LITELLM_BASE_URL=http://litellm:4000 | ||
| LITELLM_API_KEY=sk-1234 |
There was a problem hiding this comment.
Hardcoding an API key in a committed environment file is a security risk. Use a placeholder or reference an environment variable that can be set locally in a git-ignored .env file.
LITELLM_API_KEY=${LITELLM_API_KEY:-sk-placeholder}
| environment: | ||
| POSTGRES_DB: litellm | ||
| POSTGRES_USER: litellm | ||
| POSTGRES_PASSWORD: litellm_password |
There was a problem hiding this comment.
Hardcoded database password in the compose file. It is recommended to use environment variables for sensitive credentials to avoid exposing them in version control.
POSTGRES_PASSWORD: ${LITELLM_DB_PASSWORD:-litellm_password}| - ./litellm-config.yml:/app/config.yaml | ||
| environment: | ||
| # Having the Master Key hard-coded here is NOT the Best Practice | ||
| - LITELLM_MASTER_KEY=sk-1234 |
There was a problem hiding this comment.
Hardcoded LiteLLM Master Key. This should be managed via environment variables to prevent unauthorized access and secret exposure.
- LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY:-sk-1234}| - model_name: qwen3:1.7b #Can be named anything - qwen3-1.7b | ||
| litellm_params: | ||
| model: ollama/qwen3:1.7b | ||
| api_base: http://host.docker.internal:11434 |
There was a problem hiding this comment.
Using host.docker.internal points to the host machine. However, Dockerfile-litellm starts Ollama inside the deepwiki container. To allow the litellm container to reach the containerized Ollama, use the service name deepwiki instead. This also applies to line 17.
api_base: http://deepwiki:11434| - model_name: nomic-embed-text | ||
| litellm_params: | ||
| model: ollama/nomic-embed-text | ||
| api_base: http://host.docker.internal:11434 |
There was a problem hiding this comment.
As noted for the generation model, this should point to the deepwiki service name to reach the containerized Ollama instance.
api_base: http://deepwiki:11434| COPY --exclude=./api . . | ||
| RUN NODE_ENV=production npm run build |
There was a problem hiding this comment.
The build stage is missing several optimizations and configurations present in the original Dockerfile.
- Cache Invalidation: Using
COPY . .(even with excludes) causes the build to re-run whenever any file in the repository changes. It is better to copy only the necessary files for the build (e.g.,package.json,next.config.ts, etc.). - Memory Limits: The original
DockerfileincludedENV NODE_OPTIONS="--max-old-space-size=4096", which is often necessary for Next.js builds to avoid OOM errors. - Telemetry:
NEXT_TELEMETRY_DISABLED=1is missing.
|
|
||
| # Detect architecture and download appropriate Ollama version | ||
| # ARG TARGETARCH can be set at build time with --build-arg TARGETARCH=arm64 or TARGETARCH=amd64 | ||
| ARG TARGETARCH=arm64 |
There was a problem hiding this comment.
Hardcoding a default value for TARGETARCH can lead to incorrect binary downloads when building on different architectures (e.g., amd64) if the build argument is not explicitly passed. It is recommended to let the Docker build environment provide this value automatically via Buildx.
ARG TARGETARCH
| RUN ollama serve > /dev/null 2>&1 & \ | ||
| sleep 20 && \ | ||
| ollama pull nomic-embed-text && \ | ||
| ollama pull qwen3:1.7b |
There was a problem hiding this comment.
Pulling large LLM models during the Docker build process significantly increases the final image size and makes the build fragile as it depends on external registry availability. Consider using a persistent volume for models or pulling them at runtime via a startup script to keep the image lightweight.
| RUN apt-get update && apt-get install -y \ | ||
| curl \ | ||
| gnupg \ | ||
| git \ |
There was a problem hiding this comment.
This block is missing ca-certificates, which was present in the original Dockerfile. Furthermore, the custom certificate installation logic (lines 57-66 in the original Dockerfile) has been omitted. This will cause issues in environments that require custom CA bundles for network requests.
RUN apt-get update && apt-get install -y \
curl \
gnupg \
git \
ca-certificates \
1 participant
Summary:
This adds an optional LiteLLM Docker Compose setup for local LLM routing.
PR1 - Docker compose (current PR)
PR2 - Integration code (will follow)
PR3 - Documentation
Details:
This PR introduces an optional Docker Compose setup for LiteLLM, enabling a local LLM gateway that can be used alongside the existing DeepWiki-Open stack.
It is fully opt-in and does not modify the default application runtime or existing Docker configuration.
Changes
docker-compose-litellm.ymlfor standalone DeepWiki-Open + LiteLLM stackdocker-compose-litellm.envfor environment configurationDockerfile-litellmfor service containerizationlitellm-config.ymlfor runtime configurationNote: Currently
Dockerfile-litellmmirrors the existingDockerfileto keep the setup consistent. It serves as an extension point for future modifications (e.g., optional removal of Ollama dependency).Key Properties
Usage
Impact
Future Work