feat: new mutex GPU scheduling policy#2011
Conversation
|
Note Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe PR adds the ChangesGPU scheduling behavior
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Pod
participant Scheduler
participant DeviceUsageList
participant DeviceFit
Pod->>Scheduler: provide mutex and NUMA annotations
Scheduler->>DeviceUsageList: propagate NUMA binding
DeviceFit->>DeviceUsageList: apply policy-aware ordering
DeviceFit->>DeviceUsageList: inspect Used values
DeviceFit-->>Pod: return allocation or ExclusiveDeviceAllocateConflict
Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
be8e91c to
e67ea6f
Compare
Codecov Report❌ Patch coverage is
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
e67ea6f to
d11fc02
Compare
d11fc02 to
b9c45de
Compare
b9c45de to
8337c93
Compare
8337c93 to
0868126
Compare
0a74188 to
6509645
Compare
NUMA was the primary key in Less(), which broke binpack and spread when devices span NUMA nodes. Score is now primary, NUMA is the tiebreaker. Keep NUMA grouping when a pod requests numa-bind, so Fit can still accumulate a same-NUMA run and NUMA affinity is not broken. Fixes Project-HAMi#1806 Closes Project-HAMi#2010 Part of Project-HAMi#1889 Signed-off-by: mesutoezdil <mesudozdil@gmail.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
pkg/device/metax/sdevice_test.go (1)
1179-1207: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winCover the positive mutex path and assert failure results.
This only proves rejection. Add a used+idle case asserting the idle UUID is selected, and validate
wantLenregardless ofwantFit; otherwise a mutex implementation that rejects all devices, or returns partial allocations withfalse, still passes.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/device/metax/sdevice_test.go` around lines 1179 - 1207, Extend the mutex-policy tests around the existing “mutex policy rejects used device” case to include both a used device and an idle device, asserting that the idle device’s UUID is selected. Update assertions to always validate wantLen and wantDevIDs regardless of wantFit, including failure cases, so implementations that reject all devices or return partial allocations with false are caught.pkg/scheduler/scheduler.go (1)
45-45: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider moving the
NumaBindannotation key to a shared package.The scheduler now imports
pkg/device/nvidiasolely for thenvidia.NumaBindconstant. Since NUMA binding is applied across all device backends (not just NVIDIA), defining the annotation key in a vendor-specific package creates an awkward coupling. Consider moving it topkg/utilorpkg/deviceto keep the scheduler vendor-agnostic.Also applies to: 529-541, 559-559
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/scheduler/scheduler.go` at line 45, Move the NumaBind annotation constant from the NVIDIA-specific package to a shared package such as pkg/device or pkg/util, update its definition and all references including scheduler NUMA-binding logic, remove the scheduler’s nvidia import, and adjust affected tests or callers to use the shared symbol.pkg/device/biren/device.go (1)
196-196: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winCorrect mutex-conflict guard, but duplicated across every backend.
The exact same 5-line block (compute
isMutex, rejectdev.Used > 0, incrementcommon.ExclusiveDeviceAllocateConflict, log, continue) is copy-pasted identically acrossbiren,iluvatar,metax,nvidia,vastai(and, per the PR description, all 13 device backends). Consider extracting a shared helper inpkg/device/commonto avoid divergence risk as backends evolve independently.♻️ Proposed helper extraction
// pkg/device/common/common.go func RejectIfMutexConflict(reason map[string]int, pod *corev1.Pod, dev *device.DeviceUsage, isMutex bool, index int) bool { if isMutex && dev.Used > 0 { reason[ExclusiveDeviceAllocateConflict]++ klog.V(5).InfoS(ExclusiveDeviceAllocateConflict, "pod", klog.KObj(pod), "device", dev.ID, "device index", index, "used", dev.Used) return true } return false }Call sites become:
- if isMutex && dev.Used > 0 { - reason[common.ExclusiveDeviceAllocateConflict]++ - klog.V(5).InfoS(common.ExclusiveDeviceAllocateConflict, "pod", klog.KObj(pod), "device", dev.ID, "device index", i, "used", dev.Used) - continue - } + if common.RejectIfMutexConflict(reason, pod, dev, isMutex, i) { + continue + }Also applies to: 217-221
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/device/biren/device.go` at line 196, Extract the duplicated mutex-conflict guard into a shared common helper, such as RejectIfMutexConflict, accepting the reason map, pod, device usage, mutex flag, and index; have it increment ExclusiveDeviceAllocateConflict, log the conflict, and return whether allocation should be skipped. Replace the repeated isMutex/dev.Used checks and associated logging across all device backends, including biren and the other listed implementations, with this helper while preserving each backend’s existing control flow.pkg/device/amd/device_test.go (1)
529-553: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winConsider adding a multi-device mutex selection test.
All new mutex test cases (across the 8 backend files) only verify rejection when the sole candidate device is busy. None verify that, given a mix of busy and idle devices under the mutex policy,
Fitactually selects the idle one — which is the core scheduling behavior described in the PR. If this is already covered bygpu_policy_test.go's ordering tests, this can be skipped; otherwise, adding a device pair (oneUsed>0, oneUsed=0) here would close the gap.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/device/amd/device_test.go` around lines 529 - 553, Add a mutex-policy test covering mixed device availability in the relevant test table: include one busy device (Used>0) and one idle device (Used=0), request a single AMD device, and assert Fit succeeds while selecting only the idle device with the expected ID and count. Reuse the existing test symbols and expectations such as wantFit, wantLen, wantDevIDs, and wantReason, unless gpu_policy_test.go already verifies this behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/device/awsneuron/device.go`:
- Line 372: The mutex policy is bypassed in the multi-device selection path. In
the request-handling logic near isMutex and graphSelect, filter out devices with
Used > 0 before calling graphSelect when mutex scheduling is enabled, while
preserving the existing behavior for non-mutex policies and single-device
requests.
In `@pkg/device/kunlun/vdevice.go`:
- Around line 233-240: The mutex-specific filter in Fit’s graghSelect path
currently causes rejected used VXPUs to be reported as NumaNotFit. Track whether
candidates were excluded by the mutex policy, return
ExclusiveDeviceAllocateConflict when that filter prevents allocation, and
preserve NumaNotFit for genuine topology failures; update
TestKunlunVDevices_Fit_Mutex to assert the new reason.
---
Nitpick comments:
In `@pkg/device/amd/device_test.go`:
- Around line 529-553: Add a mutex-policy test covering mixed device
availability in the relevant test table: include one busy device (Used>0) and
one idle device (Used=0), request a single AMD device, and assert Fit succeeds
while selecting only the idle device with the expected ID and count. Reuse the
existing test symbols and expectations such as wantFit, wantLen, wantDevIDs, and
wantReason, unless gpu_policy_test.go already verifies this behavior.
In `@pkg/device/biren/device.go`:
- Line 196: Extract the duplicated mutex-conflict guard into a shared common
helper, such as RejectIfMutexConflict, accepting the reason map, pod, device
usage, mutex flag, and index; have it increment ExclusiveDeviceAllocateConflict,
log the conflict, and return whether allocation should be skipped. Replace the
repeated isMutex/dev.Used checks and associated logging across all device
backends, including biren and the other listed implementations, with this helper
while preserving each backend’s existing control flow.
In `@pkg/device/metax/sdevice_test.go`:
- Around line 1179-1207: Extend the mutex-policy tests around the existing
“mutex policy rejects used device” case to include both a used device and an
idle device, asserting that the idle device’s UUID is selected. Update
assertions to always validate wantLen and wantDevIDs regardless of wantFit,
including failure cases, so implementations that reject all devices or return
partial allocations with false are caught.
In `@pkg/scheduler/scheduler.go`:
- Line 45: Move the NumaBind annotation constant from the NVIDIA-specific
package to a shared package such as pkg/device or pkg/util, update its
definition and all references including scheduler NUMA-binding logic, remove the
scheduler’s nvidia import, and adjust affected tests or callers to use the
shared symbol.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 5d6d0642-b4ed-4aa2-94d5-d77cedcfb2cd
📒 Files selected for processing (33)
pkg/device/amd/device.gopkg/device/amd/device_test.gopkg/device/ascend/device.gopkg/device/ascend/device_test.gopkg/device/awsneuron/device.gopkg/device/awsneuron/device_test.gopkg/device/biren/device.gopkg/device/biren/device_test.gopkg/device/cambricon/device.gopkg/device/cambricon/device_test.gopkg/device/enflame/device.gopkg/device/enflame/device_test.gopkg/device/hygon/device.gopkg/device/hygon/device_test.gopkg/device/iluvatar/device.gopkg/device/iluvatar/device_test.gopkg/device/kunlun/device_test.gopkg/device/kunlun/vdevice.gopkg/device/metax/device.gopkg/device/metax/device_test.gopkg/device/metax/sdevice.gopkg/device/metax/sdevice_test.gopkg/device/mthreads/device.gopkg/device/mthreads/device_test.gopkg/device/nvidia/device.gopkg/device/nvidia/device_test.gopkg/device/vastai/device.gopkg/device/vastai/device_test.gopkg/scheduler/numa_sort_test.gopkg/scheduler/policy/gpu_policy.gopkg/scheduler/policy/gpu_policy_test.gopkg/scheduler/scheduler.gopkg/util/types.go
Add GPUSchedulerPolicyMutex = "mutex" constant. When a pod sets hami.io/gpu-scheduler-policy: mutex, the scheduler allocates only GPUs with no existing users (Used == 0). - Less(): mutex case sorts idle GPUs to tail for Fit() to pick first - Fit(): isMutex filter rejects devices with Used > 0 - Applied to all 13 device backends Closes Project-HAMi#2009 Part of Project-HAMi#1889 Signed-off-by: mesutoezdil <mesudozdil@gmail.com>
6509645 to
4ea20e0
Compare
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pkg/scheduler/scheduler.go (1)
762-777: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick winSort device vendors before locking to prevent deadlocks.
Iterating over
device.GetDevices()yields vendors in a non-deterministic order because it returns a Go map. If multiple pods are scheduled concurrently on a node with multiple device vendors, acquiring locks in random orders can cause livelocks or deadlocks (e.g., Pod A locks Vendor 1 then waits for Vendor 2, while Pod B locks Vendor 2 then waits for Vendor 1).Sort the vendor names to enforce a strict lock acquisition order, and release them in reverse order to ensure deterministic behavior.
🔒️ Proposed fix to acquire and release locks deterministically
-func (s *Scheduler) lockAllDevices(node *corev1.Node, pod *corev1.Pod) error { - for _, val := range device.GetDevices() { - if err := val.LockNode(node, pod); err != nil { - return err - } - } - return nil -} - -func (s *Scheduler) releaseAllDevices(node *corev1.Node, pod *corev1.Pod) { - for _, val := range device.GetDevices() { - if err := val.ReleaseNodeLock(node, pod); err != nil { - klog.ErrorS(err, "Failed to release node lock", "node", node.Name, "pod", klog.KObj(pod)) - } - } -} +func (s *Scheduler) lockAllDevices(node *corev1.Node, pod *corev1.Pod) error { + devs := device.GetDevices() + vendors := make([]string, 0, len(devs)) + for vendor := range devs { + vendors = append(vendors, vendor) + } + sort.Strings(vendors) + + for _, vendor := range vendors { + if err := devs[vendor].LockNode(node, pod); err != nil { + return err + } + } + return nil +} + +func (s *Scheduler) releaseAllDevices(node *corev1.Node, pod *corev1.Pod) { + devs := device.GetDevices() + vendors := make([]string, 0, len(devs)) + for vendor := range devs { + vendors = append(vendors, vendor) + } + sort.Strings(vendors) + + for i := len(vendors) - 1; i >= 0; i-- { + vendor := vendors[i] + if err := devs[vendor].ReleaseNodeLock(node, pod); err != nil { + klog.ErrorS(err, "Failed to release node lock", "node", node.Name, "pod", klog.KObj(pod)) + } + } +}(Note: You may need to run
goimportswith thegithub.com/Project-HAMi/HAMiprefix to pull in the"sort"package if it is not already imported.)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/scheduler/scheduler.go` around lines 762 - 777, Update lockAllDevices and releaseAllDevices to collect device vendor names, sort them deterministically, and access vendors in that order when locking. Release the same vendors in reverse sorted order, preserving the existing error logging and return behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@pkg/scheduler/scheduler.go`:
- Around line 762-777: Update lockAllDevices and releaseAllDevices to collect
device vendor names, sort them deterministically, and access vendors in that
order when locking. Release the same vendors in reverse sorted order, preserving
the existing error logging and return behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: fe6b03fd-0780-4945-949d-ec312b0c141e
📒 Files selected for processing (5)
pkg/device/ascend/device.gopkg/device/ascend/device_test.gopkg/device/nvidia/device.gopkg/scheduler/scheduler.gopkg/util/types.go
🚧 Files skipped from review as they are similar to previous changes (4)
- pkg/util/types.go
- pkg/device/ascend/device_test.go
- pkg/device/nvidia/device.go
- pkg/device/ascend/device.go
…nder mutex When the mutex policy rejects an otherwise-fitting device because it is already in use, Fit() returned the generic NumaNotFit reason, making it indistinguishable from a real topology mismatch. Signed-off-by: mesutoezdil <mesudozdil@gmail.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
pkg/device/kunlun/vdevice.go (1)
244-249: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRename loop variable to avoid shadowing the method receiver.
The loop variable
devshadows the method receiverdev *KunlunVDevices. While this doesn't cause a functional bug due to scoping, shadowing the receiver can lead to confusion. Consider renaming it todordeviceUsage.♻️ Proposed refactor
- for _, dev := range devices { - if dev.Used > 0 && FitVXPU(dev, request) { + for _, d := range devices { + if d.Used > 0 && FitVXPU(d, request) { reason[common.ExclusiveDeviceAllocateConflict]++ - klog.V(5).InfoS(common.ExclusiveDeviceAllocateConflict, "pod", klog.KObj(pod), "device", dev.ID, "used", dev.Used) + klog.V(5).InfoS(common.ExclusiveDeviceAllocateConflict, "pod", klog.KObj(pod), "device", d.ID, "used", d.Used) } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/device/kunlun/vdevice.go` around lines 244 - 249, Rename the inner loop variable in the device-conflict iteration to avoid shadowing the method receiver, updating its Used, FitVXPU, and ID references consistently while preserving the existing behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/device/kunlun/vdevice.go`:
- Line 253: Update the klog.V(5).InfoS call in the NUMA mismatch logging path to
remove the dangling "numa" key, leaving only complete key-value pairs while
preserving the existing log context.
---
Nitpick comments:
In `@pkg/device/kunlun/vdevice.go`:
- Around line 244-249: Rename the inner loop variable in the device-conflict
iteration to avoid shadowing the method receiver, updating its Used, FitVXPU,
and ID references consistently while preserving the existing behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 6c2bc4f1-40c5-4e2d-8ace-260f2dbffb06
📒 Files selected for processing (2)
pkg/device/kunlun/device_test.gopkg/device/kunlun/vdevice.go
🚧 Files skipped from review as they are similar to previous changes (1)
- pkg/device/kunlun/device_test.go
klog.InfoS expects key-value pairs; the trailing "numa" string had no matching value, causing an odd argument count. Signed-off-by: mesutoezdil <mesudozdil@gmail.com>
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
@coderabbitai full review |
✅ Action performedFull review finished. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/device/vastai/device.go`:
- Line 234: Update the mutex-policy filtering around computeBestCombination to
precompute the AIC identifiers of all used dies, using each die’s
CustomInfo["AIC"], then reject every die whose AIC is in that occupied set.
Replace the current per-die Used check while preserving existing non-mutex
behavior and handling missing AIC values safely.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 302405d7-c8a2-4c6e-9b7c-40716fbee507
📒 Files selected for processing (33)
pkg/device/amd/device.gopkg/device/amd/device_test.gopkg/device/ascend/device.gopkg/device/ascend/device_test.gopkg/device/awsneuron/device.gopkg/device/awsneuron/device_test.gopkg/device/biren/device.gopkg/device/biren/device_test.gopkg/device/cambricon/device.gopkg/device/cambricon/device_test.gopkg/device/enflame/device.gopkg/device/enflame/device_test.gopkg/device/hygon/device.gopkg/device/hygon/device_test.gopkg/device/iluvatar/device.gopkg/device/iluvatar/device_test.gopkg/device/kunlun/device_test.gopkg/device/kunlun/vdevice.gopkg/device/metax/device.gopkg/device/metax/device_test.gopkg/device/metax/sdevice.gopkg/device/metax/sdevice_test.gopkg/device/mthreads/device.gopkg/device/mthreads/device_test.gopkg/device/nvidia/device.gopkg/device/nvidia/device_test.gopkg/device/vastai/device.gopkg/device/vastai/device_test.gopkg/scheduler/numa_sort_test.gopkg/scheduler/policy/gpu_policy.gopkg/scheduler/policy/gpu_policy_test.gopkg/scheduler/scheduler.gopkg/util/types.go
In die mode a physical card (AIC) is made of several dies. The mutex guard only checked the current die's Used count, so an idle sibling die on a physically occupied card could still be allocated, breaking the whole-card exclusivity the mutex policy promises. Precompute the set of occupied AICs and reject any idle die whose AIC is already occupied. Non-mutex behavior is unchanged and missing AIC values are handled safely. Add die-mode mutex tests. Signed-off-by: mesutoezdil <mesudozdil@gmail.com>
|
If you found this review helpful, would you consider giving us a shout-out on X? Thank you for using CodeRabbit! ✅ Action performedReview finished.
|
|
/assign @archlitchi |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: archlitchi, mesutoezdil The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| klog.V(5).InfoS(common.CardTimeSlicingExhausted, "pod", klog.KObj(pod), "device", dev.ID, "count", dev.Count, "used", dev.Used) | ||
| continue | ||
| } | ||
| if isMutex && dev.Used > 0 { |
There was a problem hiding this comment.
hi @mesutoezdil I'm a bit confused about the mutex policy.
Judging from the code, the mutex scheduling policy is implemented as restricted, not best-effort. If a card is already allocated via the mutex policy, can it still be shared with other Pods?
If so, why is restricted used here? And if it cannot be shared with other Pods, why use mutex instead of just requesting a full card?
There was a problem hiding this comment.
hi @mesutoezdil I'm a bit confused about the
mutexpolicy. Judging from the code, the mutex scheduling policy is implemented asrestricted, notbest-effort. If a card is already allocated via themutexpolicy, can it still be shared with other Pods? If so, why isrestrictedused here? And if it cannot be shared with other Pods, why usemutexinstead of just requesting a full card?
hi @DSFans2014 as implemented, it's a placement-time preference for the requesting pod, not a device-level lock, it matches #2009's literal spec ("only allocates GPUs with no existing users, skipped if already in use"), but that spec never covered protecting the device from later non-mutex pods.
So today: a mutex pod won't join a busy card, but a later non-mutex pod can still join a card a mutex pod already claimed.
If true mutual exclusion is the goal, that needs a persistent reservation marker on the device (checked by every pod's Fit(), not just mutex ones). If this is the ‘new/ideal’ architecture, let’s open a new issue. WDYT?
There was a problem hiding this comment.
I got, thank you
Moreover, it would be better to also add some explanation and usage about mutex in the documentation.
There was a problem hiding this comment.
@DSFans2014 sure, opened Project-HAMi/website#625 covering both pages
3 participants
Add a
mutexGPU scheduling policy. A pod that setshami.io/gpu-scheduler-policy: mutexis only placed on GPUs that have no other users (Used == 0).Less()sorts busy GPUs first and idle GPUs to the tail, andFit()skips any device withUsed > 0under the mutex policy, across all device backends.Based on #2012 (Score-primary sort). This PR is rebased on top of #2012 and only adds the mutex code, so it does not repeat the sort fix.
Changes
pkg/util/types.go: addGPUSchedulerPolicyMutex.pkg/scheduler/policy/gpu_policy.go: mutex branch inLess().Fit()rejects used devices under the mutex policy.Fixes #2009
Part of #1889
Summary by CodeRabbit
mutexGPU scheduling policy intended to allocate only idle devices.ExclusiveDeviceAllocateConflict.