Skip to content

docs: document mutex GPU scheduler policy #625

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mesutoezdil wants to merge 2 commits into
base: master
Choose a base branch
from
Open

docs: document mutex GPU scheduler policy #625

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ac22838
docs: document mutex GPU scheduler policy
mesutoezdil Jul 21, 2026
6c75481
Merge branch 'master' into docs/mutex-gpu-policy
mesutoezdil Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions docs/developers/scheduling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,3 +175,17 @@ GPU2 Score: ((20+70)/100 + (1000+6000)/8000)) * 10 = 17.75
```

In `Spread` policy, `GPU1` is selected.

#### Mutex

Mutex gives a pod exclusive use of a GPU card. Only cards with no existing workloads (`used == 0`) are eligible; any card already in use is skipped with the `ExclusiveDeviceAllocateConflict` reason. If every card on a node is in use, the pod cannot be scheduled to that node.

Set it per pod via the annotation:

```yaml
metadata:
annotations:
hami.io/gpu-scheduler-policy: "mutex"
```

Using the same example, `GPU1` and `GPU2` both already have workloads, so neither is eligible and the pod stays pending until a fully idle card is available. A card allocated to a `mutex` pod can still be selected for other pods afterwards; to keep the card exclusive for the lifetime of the workload, request all of its resources or use it together with `use-gpuuuid` constraints.
4 changes: 2 additions & 2 deletions docs/userguide/configure.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ helm install hami hami-charts/hami --set devicePlugin.deviceMemoryScaling=5 -n k
| --- | --- | --- | --- |
| `devicePlugin.service.schedulerPort` | Integer | Scheduler webhook service nodePort. | `31998` |
| `scheduler.defaultSchedulerPolicy.nodeSchedulerPolicy` | String | GPU node scheduling policy: `"binpack"` allocates jobs to the same GPU node as much as possible. `"spread"` allocates jobs to different GPU nodes as much as possible. | `"binpack"` |
| `scheduler.defaultSchedulerPolicy.gpuSchedulerPolicy` | String | GPU scheduling policy: `"binpack"` allocates jobs to the same GPU as much as possible. `"spread"` allocates jobs to different GPUs as much as possible. | `"spread"` |
| `scheduler.defaultSchedulerPolicy.gpuSchedulerPolicy` | String | GPU scheduling policy: `"binpack"` allocates jobs to the same GPU as much as possible. `"spread"` allocates jobs to different GPUs as much as possible. `"mutex"` allocates jobs only to GPUs with no other workloads. | `"spread"` |

## Pod Configs: Annotations

Expand All @@ -80,7 +80,7 @@ helm install hami hami-charts/hami --set devicePlugin.deviceMemoryScaling=5 -n k
| `nvidia.com/nouse-gputype` | String | If set, devices allocated by this pod will NOT be in the types defined in this string. | `"Tesla V100-PCIE-32GB, NVIDIA A10"` |
| `nvidia.com/use-gputype` | String | If set, devices allocated by this pod MUST be one of the types defined in this string. | `"Tesla V100-PCIE-32GB, NVIDIA A10"` |
| `hami.io/node-scheduler-policy` | String | GPU node scheduling policy: `"binpack"` allocates the pod to used GPU nodes for execution. `"spread"` allocates the pod to different GPU nodes for execution. | `"binpack"` or `"spread"` |
| `hami.io/gpu-scheduler-policy` | String | GPU scheduling policy: `"binpack"` allocates the pod to the same GPU card for execution. `"spread"` allocates the pod to different GPU cards for execution. | `"binpack"` or `"spread"` |
| `hami.io/gpu-scheduler-policy` | String | GPU scheduling policy: `"binpack"` allocates the pod to the same GPU card for execution. `"spread"` allocates the pod to different GPU cards for execution. `"mutex"` allocates the pod only to a GPU card with no other workloads, giving it exclusive use of that card. | `"binpack"`, `"spread"` or `"mutex"` |
| `nvidia.com/vgpu-mode` | String | The type of vGPU instance this pod wishes to use. | `"hami-core"` or `"mig"` |

## Container Configs: Env
Expand Down