fix(install): set DM_DISABLE_UDEV=1 to prevent dm semaphore deadlock

`bootc install to-disk --block-setup tpm2-luks` hangs at
`cryptsetup luksOpen` due to a libdevmapper udev cookie semaphore
deadlock. libdevmapper creates SysV semaphores to synchronize with
udevd, but the container's isolated IPC namespace prevents udevd
(on the host) from seeing the semaphore. luksOpen blocks forever
on semop().

Set DM_DISABLE_UDEV=1 before LUKS operations to skip udev
synchronization for device-mapper. This is safe during installation
-- there are no concurrent dm operations, and the kernel still
creates device nodes without udev involvement. Partition device
nodes and udev_settle() are unaffected.

Confirmed with three independent tests:
- Stock bootc, default podman: HANG
- Stock bootc + DM_DISABLE_UDEV=1: PASS
- Stock bootc + --ipc=host: PASS

Fixes: #2089
Related: #421, #477
Signed-off-by: Andrew Dunn <andrew@dunn.dev>

Author: andrewdunndev

crates/lib/src/install/baseline.rs

@@ -149,6 +149,29 @@ pub(crate) fn wipefs(dev: &Utf8Path) -> Result<()> {
         .run_inherited_with_cmd_context()
 }
 
+/// Disable libdevmapper's udev synchronization for device-mapper operations.
+///
+/// libdevmapper uses SysV semaphores ("udev cookies") to coordinate with udevd.
+/// Inside a container with an isolated IPC namespace (the podman/docker default),
+/// udevd on the host cannot see the container's semaphores, causing `cryptsetup
+/// luksOpen` and `luksClose` to deadlock on `semop()`.
+///
+/// Only affects device-mapper operations (LUKS open/close). Partition device
+/// nodes created by the kernel and managed by udev are not affected, so
+/// `udev_settle()` for partition discovery continues to work normally.
+///
+/// This is safe during installation as there are no concurrent device-mapper
+/// consumers.
+#[allow(unsafe_code)]
+fn disable_dm_udev_sync() {
+    // SAFETY: bootc uses a single-threaded tokio runtime (new_current_thread).
+    // No other threads can race on environment reads. The variable is only
+    // inherited by child processes (cryptsetup, systemd-cryptenroll).
+    unsafe {
+        std::env::set_var("DM_DISABLE_UDEV", "1");
+    }
+}
+
 pub(crate) fn udev_settle() -> Result<()> {
     // There's a potential window after rereading the partition table where
     // udevd hasn't yet received updates from the kernel, settle will return
@@ -350,6 +373,15 @@ pub(crate) fn install_create_rootfs(
     let (rootdev_path, root_blockdev_kargs) = match block_setup {
         BlockSetup::Direct => (root_device.path(), None),
         BlockSetup::Tpm2Luks => {
+            // Disable libdevmapper's udev synchronization. libdevmapper uses
+            // SysV semaphores ("udev cookies") to coordinate device-mapper
+            // operations with udevd. Inside a container with an isolated IPC
+            // namespace (the podman/docker default), udevd on the host cannot
+            // see the container's semaphores, causing cryptsetup luksOpen and
+            // luksClose to deadlock on semop(). This is safe during
+            // installation as there are no concurrent device-mapper consumers.
+            disable_dm_udev_sync();
+
             let uuid = uuid::Uuid::new_v4().to_string();
             // This will be replaced via --wipe-slot=all when binding to tpm below
             let dummy_passphrase = uuid::Uuid::new_v4().to_string();