Composefs mismatch when built with buildah Github action, but not when built locally via podman

[PhysicsIsAwesome]

Hi,
an experimental proof-of-concept build pipeline of a sealed arch-bootc image fails with a composefs mismatch, when I try to install it to filesystem. Repo with build pipeline via Github action and usage of buildah-build: https://github.com/PhysicsIsAwesome/arch-bootc-sealed
Updating bootc-dev-actions/bootc-ubuntu-setup@main to get packages from the latest Ubuntu 26.04 did not help.

Surprisingly, if I build the same containerfiles locally with podman on Arch, writing it to filesystem works without issues.

Justfile for writing to filesystem:

write-to-fs $image_name="ghcr.io/physicsisawesome/arch-bootc-sealed":
    #!/usr/bin/env bash
    umount /mnt/archbootc/boot || echo "not mounted"
    umount /mnt/archbootc || echo "not mounted"
    cryptsetup open /dev/vdc2 my-luks
    mount -o subvol=@ /dev/mapper/my-luks /mnt/archbootc
    mkdir /mnt/archbootc/boot
    mount /dev/vdc1 /mnt/archbootc/boot
    rm -rf /mnt/archbootc/ || echo "ok"
    podman pull ${image_name}
    podman run --rm --privileged --pid=host -v /:/target -v /dev:/dev -v /var/lib/containers:/var/lib/containers -e RUST_LOG=debug ${image_name} bootc install to-filesystem --composefs-backend  --bootloader systemd --boot-mount-spec UUID=A775-A613 /target/mnt/archbootc
    umount /mnt/archbootc/boot
    umount /mnt/archbootc

Log for write-to-fs: https://pastebin.com/ebM6brJ7

[cgwalters]

Yes this is painful to debug right now. Basically try to get the composefs dumpfile out of both and then diff them. See the code behind the just validate-composefs-digest in this repo

[PhysicsIsAwesome]

What I don't understand is how the same Containerfiles can give different results for different build environment. Shouldn't they be the same?

With regards to the code behind just validate-composefs-digest, would you mind telling me which commands I need to run, to see what's wrong?

[Johan-Liebert1]

If you have access to the image built with GitHub Actions, then you can run bootc internals cfs oci dump <image-name>. If not, the steps are a bit convoluted. You can use the same above command get a dumpfile for you local image. For GA, you'll need to add an extra step with the above command