From ac22838b758f5ca1b43aeac4858dbd4b0a2a7bf6 Mon Sep 17 00:00:00 2001 From: mesutoezdil Date: Tue, 21 Jul 2026 10:12:00 +0200 Subject: [PATCH] docs: document mutex GPU scheduler policy Covers the mutex policy added in Project-HAMi/HAMi#2011: annotation usage in configure.md and behavior in the scheduler policy guide. Signed-off-by: mesutoezdil --- docs/developers/scheduling.md | 14 ++++++++++++++ docs/userguide/configure.md | 4 ++-- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/docs/developers/scheduling.md b/docs/developers/scheduling.md index fc01c6cb..4597384e 100644 --- a/docs/developers/scheduling.md +++ b/docs/developers/scheduling.md @@ -175,3 +175,17 @@ GPU2 Score: ((20+70)/100 + (1000+6000)/8000)) * 10 = 17.75 ``` In `Spread` policy, `GPU1` is selected. + +#### Mutex + +Mutex gives a pod exclusive use of a GPU card. Only cards with no existing workloads (`used == 0`) are eligible; any card already in use is skipped with the `ExclusiveDeviceAllocateConflict` reason. If every card on a node is in use, the pod cannot be scheduled to that node. + +Set it per pod via the annotation: + +```yaml +metadata: + annotations: + hami.io/gpu-scheduler-policy: "mutex" +``` + +Using the same example, `GPU1` and `GPU2` both already have workloads, so neither is eligible and the pod stays pending until a fully idle card is available. A card allocated to a `mutex` pod can still be selected for other pods afterwards; to keep the card exclusive for the lifetime of the workload, request all of its resources or use it together with `use-gpuuuid` constraints. diff --git a/docs/userguide/configure.md b/docs/userguide/configure.md index 1c9dc911..e3ec92e6 100644 --- a/docs/userguide/configure.md +++ b/docs/userguide/configure.md @@ -69,7 +69,7 @@ helm install hami hami-charts/hami --set devicePlugin.deviceMemoryScaling=5 -n k | --- | --- | --- | --- | | `devicePlugin.service.schedulerPort` | Integer | Scheduler webhook service nodePort. | `31998` | | `scheduler.defaultSchedulerPolicy.nodeSchedulerPolicy` | String | GPU node scheduling policy: `"binpack"` allocates jobs to the same GPU node as much as possible. `"spread"` allocates jobs to different GPU nodes as much as possible. | `"binpack"` | -| `scheduler.defaultSchedulerPolicy.gpuSchedulerPolicy` | String | GPU scheduling policy: `"binpack"` allocates jobs to the same GPU as much as possible. `"spread"` allocates jobs to different GPUs as much as possible. | `"spread"` | +| `scheduler.defaultSchedulerPolicy.gpuSchedulerPolicy` | String | GPU scheduling policy: `"binpack"` allocates jobs to the same GPU as much as possible. `"spread"` allocates jobs to different GPUs as much as possible. `"mutex"` allocates jobs only to GPUs with no other workloads. | `"spread"` | ## Pod Configs: Annotations @@ -80,7 +80,7 @@ helm install hami hami-charts/hami --set devicePlugin.deviceMemoryScaling=5 -n k | `nvidia.com/nouse-gputype` | String | If set, devices allocated by this pod will NOT be in the types defined in this string. | `"Tesla V100-PCIE-32GB, NVIDIA A10"` | | `nvidia.com/use-gputype` | String | If set, devices allocated by this pod MUST be one of the types defined in this string. | `"Tesla V100-PCIE-32GB, NVIDIA A10"` | | `hami.io/node-scheduler-policy` | String | GPU node scheduling policy: `"binpack"` allocates the pod to used GPU nodes for execution. `"spread"` allocates the pod to different GPU nodes for execution. | `"binpack"` or `"spread"` | -| `hami.io/gpu-scheduler-policy` | String | GPU scheduling policy: `"binpack"` allocates the pod to the same GPU card for execution. `"spread"` allocates the pod to different GPU cards for execution. | `"binpack"` or `"spread"` | +| `hami.io/gpu-scheduler-policy` | String | GPU scheduling policy: `"binpack"` allocates the pod to the same GPU card for execution. `"spread"` allocates the pod to different GPU cards for execution. `"mutex"` allocates the pod only to a GPU card with no other workloads, giving it exclusive use of that card. | `"binpack"`, `"spread"` or `"mutex"` | | `nvidia.com/vgpu-mode` | String | The type of vGPU instance this pod wishes to use. | `"hami-core"` or `"mig"` | ## Container Configs: Env