Skip to content

Commit 4a03de4

Browse files
stduhpfstduhpf
committed
update docs
1 parent cbf6c99 commit 4a03de4

3 files changed

Lines changed: 256 additions & 12 deletions

File tree

‎docs/rpc.md‎

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
# Building and Using the RPC Server with `stable-diffusion.cpp`
2+
3+
This guide covers how to build a version of [the RPC server from `llama.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md) that is compatible with your version of `stable-diffusion.cpp` to manage multi-backends setups. RPC allows you to offload specific model components to a remote server.
4+
5+
> **Note on Model Location:** The model files (e.g., `.safetensors` or `.gguf`) remain on the **Client** machine. The client parses the file and transmits the necessary tensor data and computational graphs to the server. The server does not need to store the model files locally.
6+
7+
## 1. Building `stable-diffusion.cpp` with RPC client
8+
9+
First, you should build the client application from source. It requires `SD_RPC=ON` to include the RPC backend to your client.
10+
11+
```bash
12+
mkdir build
13+
cd build
14+
cmake .. \
15+
-DSD_RPC=ON \
16+
# Add other build flags here (e.g., -DSD_VULKAN=ON)
17+
cmake --build . --config Release -j $(nproc)
18+
```
19+
20+
> **Note:** Ensure you add the other flags you would normally use (e.g., `-DSD_VULKAN=ON`, `-DSD_CUDA=ON`, `-DSD_HIPBLAS=ON`, or `-DGGML_METAL=ON`), for more information about building `stable-diffusion.cpp` from source, please refer to the [build.md](build.md) documentation.
21+
22+
## 2. Ensure `llama.cpp` is at the correct commit
23+
24+
`stable-diffusion.cpp`'s RPC client is designed to work with a specific version of `llama.cpp` (compatible with the `ggml` submodule) to ensure API compatibility. The commit hash for `llama.cpp` is stored in `ggml/scripts/sync-llama.last`.
25+
26+
> **Start from Root:** Perform these steps from the root of your `stable-diffusion.cpp` directory.
27+
28+
1. Read the target commit hash from the submodule tracker:
29+
30+
```bash
31+
# Linux / WSL / MacOS
32+
HASH=$(cat ggml/scripts/sync-llama.last)
33+
34+
# Windows (PowerShell)
35+
$HASH = Get-Content -Path "ggml\scripts\sync-llama.last"
36+
```
37+
38+
2. Clone `llama.cpp` at the target commit .
39+
```bash
40+
git clone https://github.com/ggml-org/llama.cpp.git
41+
cd llama.cpp
42+
git checkout $HASH
43+
```
44+
To save on download time and storage, you can use a shallow clone to download only the target commit:
45+
```bash
46+
mkdir -p llama.cpp
47+
cd llama.cpp
48+
git init
49+
git remote add origin https://github.com/ggml-org/llama.cpp.git
50+
git fetch --depth 1 origin $HASH
51+
git checkout FETCH_HEAD
52+
```
53+
54+
## 3. Build `llama.cpp` (RPC Server)
55+
56+
The RPC server acts as the worker. You must explicitly enable the **backend** (the hardware interface, such as CUDA for Nvidia, Metal for Apple Silicon, or Vulkan) when building, otherwise the server will default to using only the CPU.
57+
58+
To find the correct flags for your system, refer to the official documentation for the [`llama.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md) repository.
59+
60+
> **Crucial:** You must include the compiler flags required to satisfy the API compatibility with `stable-diffusion.cpp` (`-DGGML_MAX_NAME=128`). Without this flag, `GGML_MAX_NAME` will default to `64` for the server, and data transfers between the client and server will fail. Of course, `-DGGML_RPC` must also be enabled.
61+
>
62+
> I recommend disabling the `LLAMA_CURL` flag to avoid unnecessary dependencies, and disabling shared library builds to avoid potential conflicts.
63+
64+
> **Build Target:** We are specifically building the `rpc-server` target. This prevents the build system from compiling the entire `llama.cpp` suite (like `llama-server`), making the build significantly faster.
65+
66+
### Linux / WSL (Vulkan)
67+
68+
```bash
69+
mkdir build
70+
cd build
71+
cmake .. -DGGML_RPC=ON \
72+
-DGGML_VULKAN=ON \ # Ensure backend is enabled
73+
-DGGML_BUILD_SHARED_LIBS=OFF \
74+
-DLLAMA_CURL=OFF \
75+
-DCMAKE_C_FLAGS=-DGGML_MAX_NAME=128 \
76+
-DCMAKE_CXX_FLAGS=-DGGML_MAX_NAME=128
77+
cmake --build . --config Release --target rpc-server -j $(nproc)
78+
```
79+
80+
### macOS (Metal)
81+
82+
```bash
83+
mkdir build
84+
cd build
85+
cmake .. -DGGML_RPC=ON \
86+
-DGGML_METAL=ON \
87+
-DGGML_BUILD_SHARED_LIBS=OFF \
88+
-DLLAMA_CURL=OFF \
89+
-DCMAKE_C_FLAGS=-DGGML_MAX_NAME=128 \
90+
-DCMAKE_CXX_FLAGS=-DGGML_MAX_NAME=128
91+
cmake --build . --config Release --target rpc-server
92+
```
93+
94+
### Windows (Visual Studio 2022, Vulkan)
95+
96+
```powershell
97+
mkdir build
98+
cd build
99+
cmake .. -G "Visual Studio 17 2022" -A x64 `
100+
-DGGML_RPC=ON `
101+
-DGGML_VULKAN=ON `
102+
-DGGML_BUILD_SHARED_LIBS=OFF `
103+
-DLLAMA_CURL=OFF `
104+
-DCMAKE_C_FLAGS=-DGGML_MAX_NAME=128 `
105+
-DCMAKE_CXX_FLAGS=-DGGML_MAX_NAME=128
106+
cmake --build . --config Release --target rpc-server
107+
```
108+
109+
## 4. Usage
110+
111+
Once both applications are built, you can run the server and the client to manage your GPU allocation.
112+
113+
### Step A: Run the RPC Server
114+
115+
Start the server. It listens for connections on the default address (usually `localhost:50052`). If your server is on a different machine, ensure the server binds to the correct interface and your firewall allows the connection.
116+
117+
**On the Server :**
118+
If running on the same machine, you can use the default address:
119+
120+
```bash
121+
./rpc-server
122+
```
123+
124+
If you want to allow connections from other machines on the network:
125+
126+
```bash
127+
./rpc-server --host 0.0.0.0
128+
```
129+
130+
> **Security Warning:** The RPC server does not currently support authentication or encryption. **Only run the server on trusted local networks**. Never expose the RPC server directly to the open internet.
131+
132+
> **Drivers & Hardware:** Ensure the Server machine has the necessary drivers installed and functional (e.g., Nvidia Drivers for CUDA, Vulkan SDK, or Metal). If no devices are found, the server will simply fallback to CPU usage.
133+
134+
### Step B: Check if the client is able to connect to the server and see the available devices
135+
136+
We're assuming the server is running on your local machine, and listening on the default port `50052`. If it's running on a different machine, you can replace `localhost` with the IP address of the server.
137+
138+
**On the Client:**
139+
140+
```bash
141+
./sd-cli --rpc localhost:50052 --list-devices
142+
```
143+
144+
If the server is running and the client is able to connect, you should see `RPC0 localhost:50052` in the list of devices.
145+
146+
Example output:
147+
(Client built without GPU acceleration, two GPUs available on the server)
148+
149+
```
150+
List of available GGML devices:
151+
Name Description
152+
-------------------
153+
CPU AMD Ryzen 9 5900X 12-Core Processor
154+
RPC0 localhost:50052
155+
RPC1 localhost:50052
156+
```
157+
158+
### Step C: Run with RPC device
159+
160+
If everything is working correctly, you can now run the client while offloading some or all of the work to the RPC server.
161+
162+
Example: Setting the main backend to the RPC0 device for doing all the work on the server.
163+
164+
```bash
165+
./sd-cli -m models/sd1.5.safetensors -p "A cat" --rpc localhost:50052 --main-backend-device RPC0
166+
```
167+
168+
---
169+
170+
## 5. Scaling: Multiple RPC Servers
171+
172+
You can connect the client to multiple RPC servers simultaneously to scale out your hardware usage.
173+
174+
Example: A main machine (192.168.1.10) with 3 GPUs, with one GPU running CUDA and the other two running Vulkan, and a second machine (192.168.1.11) only one GPU.
175+
176+
**On the first machine (Running two server instances):**
177+
178+
**Terminal 1 (CUDA):**
179+
180+
```bash
181+
# Linux / WSL
182+
export CUDA_VISIBLE_DEVICES=0
183+
cd ./build_cuda/bin/Release
184+
./rpc-server --host 0.0.0.0
185+
186+
# Windows PowerShell
187+
$env:CUDA_VISIBLE_DEVICES="0"
188+
cd .\build_cuda\bin\Release
189+
./rpc-server --host 0.0.0.0
190+
```
191+
192+
**Terminal 2 (Vulkan):**
193+
194+
```bash
195+
cd ./build_vulkan/bin/Release
196+
# ignore the first GPU (used by CUDA server)
197+
./rpc-server --host 0.0.0.0 --port 50053 -d Vulkan1,Vulkan2
198+
```
199+
200+
**On the second machine:**
201+
202+
```bash
203+
cd ./build/bin/Release
204+
./rpc-server --host 0.0.0.0
205+
```
206+
207+
**On the Client:**
208+
Pass multiple server addresses separated by commas.
209+
210+
```bash
211+
./sd-cli --rpc 192.168.1.10:50052,192.168.1.10:50053,192.168.1.11:50052 --list-devices
212+
```
213+
214+
The client will map these servers to sequential device IDs (e.g., RPC0 from the first server, RPC2, RPC3 from the second, and RPC4 from the third). With this setup, you could for example use RPC0 for the main backend, RPC1 and RPC2 for the text encoders, and RPC3 for the VAE.
215+
216+
---
217+
218+
## 6. Performance Considerations
219+
220+
RPC performance is heavily dependent on network bandwidth, as large weights and activations must be transferred back and forth over the network, especially for large models, or when using high resolutions. For best results, ensure your network connection is stable and has sufficient bandwidth (>1Gbps recommended).

‎examples/cli/README.md‎

Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,10 @@ Context Options:
4343
--high-noise-diffusion-model <string> path to the standalone high noise diffusion model
4444
--uncond-diffusion-model <string> path to the standalone unconditional diffusion model, currently used by
4545
Ideogram4 CFG
46+
--embeddings-connectors <string> path to LTXAV embeddings connectors
4647
--vae <string> path to standalone vae model
48+
--vae-format <string> VAE latent format override: auto, flux, sd3, or flux2 (default: auto)
49+
--audio-vae <string> path to standalone LTX audio vae model
4750
--taesd <string> path to taesd. Using Tiny AutoEncoder for fast decoding (low quality)
4851
--tae <string> alias of --taesd
4952
--control-net <string> path to control net model
@@ -53,12 +56,18 @@ Context Options:
5356
--tensor-type-rules <string> weight type per tensor pattern (example: "^vae\.=f16,model\.=q8_0")
5457
--photo-maker <string> path to PHOTOMAKER model
5558
--upscale-model <string> path to esrgan model.
59+
--backend <string> runtime backend assignment, e.g. cpu or clip=cpu,vae=cuda0,diffusion=vulkan0
60+
--params-backend <string> parameter backend assignment, e.g. cpu or diffusion=cpu,clip=cpu
61+
--rpc-servers <string> comma-separated list of RPC servers to connect to for offloading, in the
62+
format host:port, e.g. localhost:50052,192.168.1.3:50052
5663
-t, --threads <int> number of threads to use during computation (default: -1). If threads <= 0,
5764
then threads will be set to the number of CPU physical cores
5865
--chroma-t5-mask-pad <int> t5 mask pad size of chroma
5966
--max-vram <float> maximum VRAM budget in GiB for graph-cut segmented execution. 0 disables
6067
graph splitting; a negative value auto-detects free VRAM, sparing the
6168
specified value (e.g. -0.5 will keep at least 0.5 GiB free)
69+
--stream-layers enable residency+prefetch streaming on top of --max-vram (no effect without
70+
--max-vram; defaults to false)
6271
--force-sdxl-vae-conv-scale force use of conv scale on sdxl vae
6372
--offload-to-cpu place the weights in RAM to save VRAM, and automatically load them into VRAM
6473
when needed
@@ -109,7 +118,8 @@ Generation Options:
109118
--extra-sample-args <string> extra sampler/scheduler/guidance args, key=value list. APG supports apg_eta,
110119
apg_momentum, apg_norm_threshold, apg_norm_threshold_smoothing; SLG supports
111120
slg_uncond; lcm supports noise_clip_std, noise_scale_start, noise_scale_end;
112-
ltx2 supports max_shift, base_shift, stretch, terminal; euler_ge supports gamma
121+
ltx2 supports max_shift, base_shift, stretch, terminal; euler_ge supports
122+
gamma
113123
--extra-tiling-args <string> extra VAE tiling args, key=value list. LTX video VAE supports
114124
temporal_tile_frames (default: 4), temporal_tile_overlap (default: 1)
115125
-H, --height <int> image height, in pixel space (default: 512)
@@ -153,7 +163,7 @@ Generation Options:
153163
--high-noise-eta <float> (high noise) noise multiplier (default: 0 for ddim_trailing, tcd,
154164
res_multistep and res_2s; 1 for euler_a, er_sde and dpm++2s_a)
155165
--strength <float> strength for noising/unnoising (default: 0.75)
156-
--pm-style-strength <float>
166+
--pm-style-strength <float>
157167
--control-strength <float> strength to apply Control Net (default: 0.9). 1.0 corresponds to full
158168
destruction of information in init image
159169
--moe-boundary <float> timestep boundary for Wan2.2 MoE model. (default: 0.875). Only enabled if
@@ -172,13 +182,15 @@ Generation Options:
172182
-s, --seed RNG seed (default: 42, use random seed for < 0)
173183
--sampling-method sampling method, one of [euler, euler_a, heun, dpm2, dpm++2s_a, dpm++2m,
174184
dpm++2mv2, ipndm, ipndm_v, lcm, ddim_trailing, tcd, res_multistep, res_2s,
175-
er_sde, euler_cfg_pp, euler_a_cfg_pp] (default: euler for Flux/SD3/Wan, euler_a otherwise)
185+
er_sde, euler_cfg_pp, euler_a_cfg_pp](default: euler for Flux/SD3/Wan,
186+
euler_a otherwise)
176187
--high-noise-sampling-method (high noise) sampling method, one of [euler, euler_a, heun, dpm2, dpm++2s_a,
177188
dpm++2m, dpm++2mv2, ipndm, ipndm_v, lcm, ddim_trailing, tcd, res_multistep,
178-
res_2s, er_sde, euler_cfg_pp, euler_a_cfg_pp] default: euler for Flux/SD3/Wan, euler_a otherwise
189+
res_2s, er_sde, euler_cfg_pp, euler_a_cfg_pp] default: euler for
190+
Flux/SD3/Wan, euler_a otherwise
179191
--scheduler denoiser sigma scheduler, one of [discrete, karras, exponential, ays, gits,
180-
smoothstep, sgm_uniform, simple, kl_optimal, lcm, bong_tangent, ltx2], default:
181-
model-specific
192+
smoothstep, sgm_uniform, simple, kl_optimal, lcm, bong_tangent, ltx2],
193+
default: model-specific
182194
--sigmas custom sigma values for the sampler, comma-separated (e.g.,
183195
"14.61,7.8,3.5,0.0").
184196
--hires-sigmas custom sigma values for the highres fix second pass, comma-separated (e.g.,

0 commit comments

Comments
 (0)