docs(tutorials): add Lab 11 — Run SGLang on HAMi GPU Shares#654
docs(tutorials): add Lab 11 — Run SGLang on HAMi GPU Shares#654rudrakshkarpe wants to merge 3 commits into
Conversation
Add a new intermediate lab that installs HAMi on an existing NVIDIA GPU cluster and schedules an SGLang inference service on a HAMi GPU share. The lab is the SGLang counterpart to Lab 6 (vLLM): it walks through the HAMi install, deploys Qwen3-1.7B with nvidia.com/gpu, nvidia.com/gpumem and nvidia.com/gpucores caps, tests the OpenAI-compatible API, and verifies that the gpumem cap is enforced inside the Pod. All commands and outputs are captured from a verification run on a kind cluster backed by a single NVIDIA H100 80GB. Addresses Project-HAMi#528 Co-authored-by: Shivay Lamba <19529592+shivaylamba@users.noreply.github.com> Signed-off-by: rudrakshkarpe <rudraksh.karpe@gmail.com>
Add the labs/hami-sglang entry to sidebars-tutorials.js so the new lab appears in the Labs category navigation. Co-authored-by: Shivay Lamba <19529592+shivaylamba@users.noreply.github.com> Signed-off-by: rudrakshkarpe <rudraksh.karpe@gmail.com>
|
@rudrakshkarpe: The label(s) DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
✅ Deploy Preview for project-hami ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: rudrakshkarpe The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
📝 WalkthroughWalkthroughAdds Lab 11 to the tutorials sidebar and introduces a complete guide for installing HAMi, deploying SGLang with GPU-share limits, testing its OpenAI-compatible API, validating enforcement, troubleshooting, and cleanup. ChangesHAMi SGLang lab
Estimated code review effort: 2 (Simple) | ~10 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant SGLangService
participant SGLangPod
participant HAMiScheduler
Client->>SGLangService: request OpenAI-compatible API
SGLangService->>SGLangPod: forward request
HAMiScheduler->>SGLangPod: apply GPU-share limits
SGLangPod-->>SGLangService: return inference response
SGLangService-->>Client: return response
Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The "Next Steps" section linked to ./hami-kitops (Lab 12), which does not exist on master yet, so `docusaurus build` failed its broken-link check. Point at the tracking issue (website#561) instead, keeping this lab self-contained and independently buildable. Co-authored-by: Shivay Lamba <19529592+shivaylamba@users.noreply.github.com> Signed-off-by: rudrakshkarpe <rudraksh.karpe@gmail.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tutorials/labs/hami-sglang.md`:
- Around line 526-530: Update the cleanup instructions around the kubectl label
command to remove gpu=on only when this lab added it. Have users record the node
label’s prior state before applying changes, and preserve the label when it
already existed so other workloads or DaemonSets are unaffected.
- Around line 137-143: The HAMi setup instructions must explicitly prevent
coexistence with the vendor NVIDIA device plugin. Update the surrounding
installation section to instruct users to disable or remove the NVIDIA device
plugin while retaining NVIDIA drivers/runtime support, and add cleanup
instructions for restoring or reinstalling the vendor plugin after HAMi is
removed.
- Around line 72-73: Update the GPU prerequisite in the lab guide to remove the
claim that an A10 works with the current configuration; either require GPUs with
at least 25,000 MiB of VRAM or document an A10-specific lower gpumem value
matching the manifest requirements.
- Around line 274-275: Update the SGLang image reference in the Kubernetes
example to the exact tested version tag or immutable digest instead of
lmsysorg/sglang:latest, preserving the existing imagePullPolicy unless required
otherwise. Apply the same pinned reference to the corresponding English example
and Chinese mirror if present.
- Around line 20-31: Update the HAMi verification sections around the in-pod
nvidia-smi checks and the corresponding evidence summary to describe soft
CUDA/NVML-level isolation rather than a hard GPU memory cap. Add an over-quota
CUDA allocation test that requests memory beyond the configured 25000MiB limit
and explicitly expects CUDA_ERROR_OUT_OF_MEMORY, while retaining the quota
visibility check as supporting evidence. Apply the same terminology and
validation behavior to the sections covering the later verification and
conclusion steps.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: f86ad967-4005-4f92-ad3c-a21ad6fbe64c
📒 Files selected for processing (2)
sidebars-tutorials.jstutorials/labs/hami-sglang.md
| This lab demonstrates how to install HAMi on a Kubernetes cluster that already has NVIDIA GPUs, and use HAMi to schedule an [SGLang](https://github.com/sgl-project/sglang) inference service. Upon completion, you will have an OpenAI-compatible model service that can be verified through `/v1/models` and `/v1/chat/completions`, with HAMi enforcing GPU memory and compute caps inside the Pod. | ||
|
|
||
| This guide is modeled after [Lab 6: Run vLLM on HAMi GPU Shares](./hami-vllm). The steps are not tied to a specific cloud vendor. As long as your Kubernetes cluster has available NVIDIA GPUs, NVIDIA drivers, and container runtime support, you can reproduce the same setup. | ||
|
|
||
| ## Learning Objectives | ||
|
|
||
| - Verify that an existing GPU Kubernetes cluster meets the prerequisites for HAMi and SGLang | ||
| - Install HAMi scheduler and device plugin | ||
| - Add labels required by the HAMi DaemonSet to GPU nodes | ||
| - Run SGLang using HAMi's `nvidia.com/gpu`, `nvidia.com/gpumem`, and `nvidia.com/gpucores` resources | ||
| - Test the SGLang OpenAI-compatible API via port forwarding | ||
| - Confirm that `nvidia.com/gpumem` is enforced inside the SGLang Pod |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== file line count and relevant snippets ==="
wc -l tutorials/labs/hami-sglang.md
printf '\n--- lines 1-50 ---\n'
sed -n '1,70p' tutorials/labs/hami-sglang.md
printf '\n--- lines 310-360 ---\n'
sed -n '310,360p' tutorials/labs/hami-sglang.md
printf '\n--- lines 440-500 ---\n'
sed -n '440,500p' tutorials/labs/hami-sglang.md
printf '\n--- lines 515-550 ---\n'
sed -n '515,550p' tutorials/labs/hami-sglang.md
printf '\n=== search for relevant terms ===\n'
rg -n "hard|quota|25000MiB|nvidia-smi|CUDA OOM|soft|HAMi|nvidia.com/gpumem|over-limit|allocation|enforced" tutorials/labs/hami-sglang.mdRepository: Project-HAMi/website
Length of output: 13544
🌐 Web query:
HAMi FAQ soft isolation CUDA API nvidia-smi quota
💡 Result:
HAMi provides GPU virtualization and resource isolation for Kubernetes using a user-space library interception mechanism, rather than kernel-level drivers or hardware partitions [1][2][3]. Core Mechanisms and Isolation HAMi injects a shared library, libvgpu.so, into containers via /etc/ld.so.preload [1][4][2]. This library intercepts CUDA and NVML API calls to enforce limits at the runtime level [2]. 1. Memory Isolation: HAMi enforces a hard limit on VRAM usage [2][5]. When an application attempts to allocate memory, libvgpu.so checks the request against the nvidia.com/gpumem quota [2][6]. If the request exceeds the remaining budget, the library returns a CUDA_ERROR_OUT_OF_MEMORY to the application, preventing it from exceeding its allocated slice [1][2][6]. 2. Compute Isolation: HAMi applies a soft compute limit using a token-bucket throttle mechanism embedded within libvgpu.so [1][4][5]. This controls the rate of kernel launch calls to cap compute usage at the requested percentage (nvidia.com/gpucores), acting as a rate limit rather than hardware-enforced isolation [2][5]. Interaction with nvidia-smi and CUDA - nvidia-smi: Because libvgpu.so intercepts NVML calls (e.g., nvmlDeviceGetMemoryInfo), nvidia-smi inside the container reports the virtualized quota (the HAMi limit) rather than the physical VRAM of the GPU [1][4][2]. The host-side nvidia-smi continues to report the physical memory and actual utilization [1][2]. - Bypassing Isolation: Limits may fail to enforce if the application bypasses the standard CUDA/NVML libraries, runs in Docker-in-Docker, or if the environment variable CUDA_DISABLE_CONTROL is set to true [1][4][3]. Comparison to Hardware Solutions Unlike NVIDIA MIG, which creates hardware-enforced partitions at the GPU engine level, HAMi’s memory enforcement is enforced at the CUDA API layer, and compute enforcement is a software-based throttle [1][4][5]. This makes HAMi flexible, allowing dynamic reconfiguration without node drains, but it provides "best-effort" multi-tenant noise isolation compared to the "strong" isolation of hardware-based methods [1][4].
Citations:
- 1: https://project-hami.io/docs/faq
- 2: https://project-hami.io/docs/core-concepts/gpu-virtualization
- 3: [docs/faq] Collect real user questions from GitHub issues and community channels #415
- 4: https://project-hami.io/docs/next/faq
- 5: https://asleekgeek.com/articles/gpu-sharing-decision-tree
- 6: https://project-hami.io/tutorials/labs/gpu-partitioning
Clarify HAMi’s soft isolation instead of showing only quota reporting.
HAMi enforces limits at the CUDA/NVML interception layer, so in-pod nvidia-smi showing 25000MiB proves the visible quota; it does not demonstrate that over-quota allocations fail. Add an over-limit allocation step expecting CUDA_ERROR_OUT_OF_MEMORY, and replace “Hard GPU memory cap” / “key evidence” with wording that reflects CUDA-level enforcement.
Also applies to lines 335-344, 466-490, and 532-540.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tutorials/labs/hami-sglang.md` around lines 20 - 31, Update the HAMi
verification sections around the in-pod nvidia-smi checks and the corresponding
evidence summary to describe soft CUDA/NVML-level isolation rather than a hard
GPU memory cap. Add an over-quota CUDA allocation test that requests memory
beyond the configured 25000MiB limit and explicitly expects
CUDA_ERROR_OUT_OF_MEMORY, while retaining the quota visibility check as
supporting evidence. Apply the same terminology and validation behavior to the
sections covering the later verification and conclusion steps.
Source: MCP tools
| - A working Kubernetes cluster | ||
| - At least 1 NVIDIA GPU node with enough free memory for the model (this guide uses a single NVIDIA H100 80GB; an A10/L40S-class GPU also works if you keep the small model and `gpumem` values) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Fix the A10 compatibility claim.
The manifest later requests nvidia.com/gpumem: "25000", but an NVIDIA A10 has 24 GB of memory; HAMi rejects requests that exceed available physical VRAM. Restrict this lab to GPUs with at least 25,000 MiB or provide a lower A10-specific memory value. (nvidia.com)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tutorials/labs/hami-sglang.md` around lines 72 - 73, Update the GPU
prerequisite in the lab guide to remove the claim that an A10 works with the
current configuration; either require GPUs with at least 25,000 MiB of VRAM or
document an A10-specific lower gpumem value matching the manifest requirements.
Source: MCP tools
| If the cluster already has a vendor NVIDIA device plugin installed, you may already see `nvidia.com/gpu`. After installing HAMi, `nvidia.com/gpu` becomes the number of vGPUs exposed by HAMi. | ||
|
|
||
| HAMi's device plugin matches `gpu=on` when managed node selectors are enabled. Label your GPU nodes: | ||
|
|
||
| ```bash | ||
| kubectl label node <gpu-node-name> gpu=on --overwrite | ||
| ``` |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Do not run HAMi alongside the existing NVIDIA device plugin.
The current wording treats an existing nvidia.com/gpu provider as harmless, but HAMi’s device plugin and NVIDIA’s official device plugin must not coexist because both register the same resource. Add an explicit migration step to disable/remove the vendor device plugin while retaining drivers/runtime support, and document how to restore it during cleanup. (project-hami.io)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tutorials/labs/hami-sglang.md` around lines 137 - 143, The HAMi setup
instructions must explicitly prevent coexistence with the vendor NVIDIA device
plugin. Update the surrounding installation section to instruct users to disable
or remove the NVIDIA device plugin while retaining NVIDIA drivers/runtime
support, and add cleanup instructions for restoring or reinstalling the vendor
plugin after HAMi is removed.
Source: MCP tools
| image: lmsysorg/sglang:latest | ||
| imagePullPolicy: IfNotPresent |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file =="
git ls-files | rg '(^|/)hami-sglang\.md$|tutorials|labs' || true
echo "== relevant lines =="
if [ -f tutorials/labs/hami-sglang.md ]; then
nl -ba tutorials/labs/hami-sglang.md | sed -n '230,300p'
fi
echo "== image references in file =="
if [ -f tutorials/labs/hami-sglang.md ]; then
rg -n 'image:|imagePullPolicy|latest|sha256:|digest' tutorials/labs/hami-sglang.md
fi
echo "== surrounding docs config/sidebar =="
fd -a 'sidebar\.js|sidebar\.ts|sidebars\.js|sidebars\.ts' . -t f
rg -l 'hami-sglang|SGLang|hami' . || trueRepository: Project-HAMi/website
Length of output: 4094
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== relevant lines =="
awk 'NR>=230 && NR<=320 { printf "%5d\t%s\n", NR, $0 }' tutorials/labs/hami-sglang.md
echo "== image references =="
grep -En 'image:|imagePullPolicy|latest|sha256:|digest|SGLang|sglang' tutorials/labs/hami-sglang.md || true
echo "== translate source references and Chinese mirror =="
for f in $(git ls-files | grep -E '(^|/)hami-sglang\.md$|Chinese translation'); do
echo "-- $f"
grep -En 'image:|imagePullPolicy|latest|sha256:|digest|LMSYS|SGLang|sglang' "$f" || true
doneRepository: Project-HAMi/website
Length of output: 12428
Pin the SGLang image to a concrete version or digest.
lmsysorg/sglang:latest with IfNotPresent allows cached images to be reused and fresh nodes to resolve latest independently, so the captured lab outputs may not match what later runs execute. Use the exact tested tag/digest in tutorials/labs/hami-sglang.md and update the English example plus its Chinese mirror if present.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tutorials/labs/hami-sglang.md` around lines 274 - 275, Update the SGLang
image reference in the Kubernetes example to the exact tested version tag or
immutable digest instead of lmsysorg/sglang:latest, preserving the existing
imagePullPolicy unless required otherwise. Apply the same pinned reference to
the corresponding English example and Chinese mirror if present.
Source: MCP tools
| Optional label cleanup: | ||
|
|
||
| ```bash | ||
| kubectl label node <gpu-node-name> gpu- | ||
| ``` |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Only remove gpu=on if this lab added it.
On an existing cluster, the label may be used by other workloads or DaemonSets. Record its prior state and avoid unconditionally deleting it during cleanup.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tutorials/labs/hami-sglang.md` around lines 526 - 530, Update the cleanup
instructions around the kubectl label command to remove gpu=on only when this
lab added it. Have users record the node label’s prior state before applying
changes, and preserve the label when it already existed so other workloads or
DaemonSets are unaffected.
1 participant
What type of PR is this?
/kind documentation
What this PR does / why we need it:
Adds Lab 11: Run SGLang on HAMi GPU Shares — a new intermediate tutorial that installs HAMi on an existing NVIDIA GPU cluster and schedules an SGLang inference service on a HAMi GPU share. It is the SGLang counterpart to the existing Lab 6 (vLLM), following the same structure so the two read as a pair.
The lab walks through:
gpu=on)nvidia.com/gpu,nvidia.com/gpumem(25000 MiB), andnvidia.com/gpucores(30%)/v1/models,/v1/chat/completions) via port-forwardgpumemcap is enforced inside the Pod (nvidia-smishows the ~25000 MiB slice while the host still reports the full 81559 MiB H100)Manifests are inline (self-contained
kubectl apply -f -blocks), and every command/output is captured from a verification run on a kind cluster backed by a single NVIDIA H100 80GB (verified: 2026-07-23).Also adds the sidebar entry in
sidebars-tutorials.js. The overview page picks the lab card up automatically from frontmatter viaLabCardGridAuto.Which issue(s) this PR fixes:
Fixes #528
Checklist:
npm run lintandnpm run format:checkpassnpm run buildsucceeds for bothenandzhgit commit -s)Summary by CodeRabbit