fix(install): set DM_DISABLE_UDEV=1 in global_init as defense-in-depth for dm semaphore deadlock

Move DM_DISABLE_UDEV=1 from a standalone function in baseline.rs to
global_init() in cli.rs, alongside the existing HOME workaround. This
is defense-in-depth for the IPC namespace semaphore deadlock that causes
cryptsetup luksOpen/luksClose to hang inside containers with isolated
IPC namespaces.

The primary fix is to run the install container with --ipc=host, which
shares the host IPC namespace and allows libdevmapper's udev cookie
semaphores to reach udevd. The env var catches cases where IPC sharing
is not configured.

Fixes: #2089
Related: #421, #477

Signed-off-by: Andrew Dunn <andrew@dunn.dev>
AI-Assisted: yes
AI-Tools: GitLab Duo, OpenCode

Author: andrewdunndev

crates/lib/src/cli.rs

@@ -1493,6 +1493,15 @@ pub fn global_init() -> Result<()> {
             std::env::set_var("HOME", "/root");
         }
     }
+    // Disable libdevmapper's udev synchronization. Inside a container with an
+    // isolated IPC namespace (the podman/docker default), udevd on the host
+    // cannot see the container's semaphores, causing cryptsetup luksOpen and
+    // luksClose to deadlock on semop(). This is a defense-in-depth measure;
+    // the primary fix is to run the install container with --ipc=host.
+    // SAFETY: Called early in main() before any threads are spawned.
+    unsafe {
+        std::env::set_var("DM_DISABLE_UDEV", "1");
+    }
     Ok(())
 }
 

crates/lib/src/install/baseline.rs

@@ -23,12 +23,12 @@ use clap::ValueEnum;
 use fn_error_context::context;
 use serde::{Deserialize, Serialize};
 
+use super::config::Filesystem;
 use super::MountSpec;
-use super::RUN_BOOTC;
-use super::RW_KARG;
 use super::RootSetup;
 use super::State;
-use super::config::Filesystem;
+use super::RUN_BOOTC;
+use super::RW_KARG;
 use crate::task::Task;
 use bootc_kernel_cmdline::utf8::Cmdline;
 #[cfg(feature = "install-to-disk")]