What
A new hands-on lab under tutorials/labs/ showing how to run KServe on top of HAMi vGPU, so multiple InferenceServices share the same physical GPU.
Suggested outline
- Prerequisites: cluster with HAMi installed (reuse the setup from the existing labs), KServe installed in raw deployment or Serverless mode
- Deploy an InferenceService with HAMi resource limits (
nvidia.com/gpu, nvidia.com/gpumem, nvidia.com/gpucores) on the predictor pod spec
- Deploy a second InferenceService on the same GPU to demonstrate sharing
- Verify isolation: check memory limits inside the pods with nvidia-smi and send inference requests to both services
- Optional: autoscaling behavior of KServe combined with GPU sharing
- Cleanup
Why
KServe is one of the most common model serving layers on Kubernetes, and GPU sharing for many small InferenceServices is exactly the HAMi use case. We already cover vLLM, SGLang is proposed in #528; KServe fills the serving-platform gap.
The lab would follow the format of the existing labs (hami-vllm.md, kueue-hami-vgpu.md), with an English version and a zh mirror.
I plan to work on this.
What
A new hands-on lab under
tutorials/labs/showing how to run KServe on top of HAMi vGPU, so multiple InferenceServices share the same physical GPU.Suggested outline
nvidia.com/gpu,nvidia.com/gpumem,nvidia.com/gpucores) on the predictor pod specWhy
KServe is one of the most common model serving layers on Kubernetes, and GPU sharing for many small InferenceServices is exactly the HAMi use case. We already cover vLLM, SGLang is proposed in #528; KServe fills the serving-platform gap.
The lab would follow the format of the existing labs (
hami-vllm.md,kueue-hami-vgpu.md), with an English version and a zh mirror.I plan to work on this.