`compile_example` hangs when called concurrently from parallel tests

[basil]

When multiple test threads call compile_example concurrently, each invocation spawns a separate cargo build --example subprocess. These subprocesses all attempt to acquire an exclusive flock on the build directory (Filesystem::open_rw_exclusive_create), and the parent cargo test process holds a lock on the same directory that won't be released until the tests complete.

Process tree when hung

      214795   /usr/bin/python3 -sP /usr/bin/fedpkg --release f43 local
        214821   rpmbuild --define _sourcedir src/fedora/rust-ptools --define _specdir ...
          217222   /bin/sh -e /var/tmp/rpm-tmp.oVUWqh
            217223   /usr/bin/cargo test -j32 -Z avoid-dev-deps --profile rpm --no-fail-fast
              218505    src/fedora/rust-ptools/rust-ptools-0.2.21-build/ptools-0.2.21/target/rpm/deps/pargs_penv_test-108be
                218512   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example pargs_penv
                218514   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example pargs_penv
                218515   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example pargs_penv
                218517   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example pargs_penv
                218518   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example penv_setenv
                218520   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example pargs_penv
                218521   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example pargs_penv
                218522   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example pargs_penv
                218523   /usr/bin/cargo build --message-format=json --target x86_64-unknown-linux-gnu --release --example penv_setenv

Stack trace of blocked subprocess

0x00007f2b53d7a67b flock+0xb
0x00005619ac359db1 <cargo::util::flock::Filesystem>::open_rw_exclusive_create::<alloc::string::String>::{closure#1}+0x11
0x00005619ac690069 cargo::util::flock::acquire+0x269
0x00005619abf95237 <cargo::util::flock::Filesystem>::open_rw_exclusive_create::<&str>+0xd7
0x00005619ac419d27 <cargo::core::compiler::layout::Layout>::new+0x9f7
0x00005619ac40bb2d <cargo::core::compiler::build_runner::BuildRunner>::prepare_units+0x2bd
0x00005619ac413216 <cargo::core::compiler::build_runner::BuildRunner>::compile+0x4d6
0x00005619ac577ca0 cargo::ops::cargo_compile::compile_ws+0xc90
0x00005619ac57c391 cargo::ops::cargo_compile::compile_with_exec+0x51
0x00005619ac583abb cargo::ops::cargo_compile::compile+0x5b
0x00005619ac90f929 cargo::commands::build::exec+0x4b9
0x00005619ac8e69df <cargo::cli::Exec>::exec+0x55f
0x00005619ac8e304f cargo::main+0x37ef
0x00005619ac8be593 std::sys::backtrace::__rust_begin_short_backtrace::<fn(), ()>+0x3
0x00005619ac8d28a9 std::rt::lang_start::<()>::{closure#0}+0x9
0x00005619ace65ca6 std::rt::lang_start_internal::h31d4445a63e30e2c+0x4b6
0x00005619ac95271c main+0x2c
0x00007f2b53c995b5 __libc_start_call_main+0x75
0x00007f2b53c99668 __libc_start_main@@GLIBC_2.34+0x88
0x00005619abef34c5 _start+0x25

Workaround

Use compile_examples (plural) in a LazyLock to build all examples with a single cargo build invocation, avoiding concurrent lock acquisition:

static EXAMPLES: LazyLock<HashMap<String, PathBuf>> = LazyLock::new(|| {
    snapbox::cmd::compile_examples(["-q"])
        .expect("failed to compile examples")
        .map(|(name, path)| (name, path.expect("failed to compile example")))
        .collect()
});

[epage]

the parent cargo test process holds a lock on the same directory that won't be released until the tests complete.

cargo test shouldn't be holding a lock on the build-dir or the target-dir. If you can make compile_examples work, then that is a sign that this isn't true.

Either

• Things are just really slow when serialized that it appears hung: while serializing the example builds will slow things down, the dependencies should be reused and so this should still be relatively quick
• You are hitting a race condition in Cargo's locking.

What version of cargo are you using for this? We had some race conditions in past versions.

[basil]

I was using cargo 1.94.0. I didn't dig too deeply into the root cause because I was able to find a workaround quickly in basil/ptools@b17fa9d. I'm afraid I don't have more time to spend on this.