Handle transient shared-memory directory state during named mutex open/create

[Copilot]

MutexTests.CrossProcess_NamedMutex_ConcurrentCreateOrOpen was failing in CI on Unix when one process observed /tmp/.dotnet*/shm between directory creation and final ownership/permission normalization. This change makes named mutex shared-memory directory validation tolerate that transient state instead of failing the open/create path.

• Directory race handling

  • Extend SharedMemoryHelpers.EnsureDirectoryExists with the same short transient retry model already used for shared-memory files.
  • Retry when stat/validation sees a directory before the creating process has finished applying the expected owner or mode.

• Failure modes addressed

  • User-scoped directories temporarily reported as owned by the wrong UID.
  • Non-user-scoped directories temporarily reported without the expected owner permissions.

• Scope

  • No behavioral change to the steady state validation rules.
  • The change only relaxes the transient handoff window during concurrent cross-process creation/open.

if (error.Error == Interop.Error.ENOENT && RetryOnTransientPermissionFailure())
{
    continue;
}

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[jkotas]

@copilot Are you sure createIfNotExist condition is right here?

[Copilot]

You're right; that condition was too strict for the open path. EnsureDirectoryExists(..., createIfNotExist: false) still returns false immediately when the directory is actually missing, but it also needs to tolerate the transient owner/mode window when another process has already created the directory and is finishing normalization. I removed the createIfNotExist gate in 18f567c so only that transient validation case retries; real failures still throw after the bounded retry window.

[jkotas]

@copilot Simplify the code flow here, like:

if (statResult != 0)
{
...
}

// If the path exists, check that it's a directory
if ((fileStatus.Mode & Interop.Sys.FileTypes.S_IFDIR) == 0)
{
    throw new IOException(SR.Format(SR.IO_SharedMemory_PathExistsButNotDirectory, directoryPath));
}

[Copilot]

Addressed in 0486326. The statResult != 0 path is now handled separately from the non-directory check.

[jkotas]

@copilot Change the logic to be possitive everywhere

if (RetryOnTransientPermissionFailure())
{
    continue;
}

throw new ....

[Copilot]

Addressed in 2e680d1. I flipped the remaining retry sites in EnsureDirectoryExists to the positive if (RetryOnTransientPermissionFailure()) { continue; } flow for consistency.