bootc switch/upgrade fails with 1.5.1 (and later) when remote registry doesn't include `Content-Length`

[ver4a]

(Edited by @cgwalters )

The core issue here appears when a registry doesn't include a Content-Length: HTTP header for GetBlob requests. Most registries do, but apparently some don't (and they're arguably a bit buggy) but we must work with them.

Original issue text

Reproducer:

1. Copy image using skopeo from docker transport to containers-storage:

skopeo copy docker://quay.io/fedora-ostree-desktops/kinoite:42 containers-storage:registry.uncontrol.me/ver4a/onc-kde:42-upstream

2. Copy the image back into a registry:

skopeo copy containers-storage:registry.uncontrol.me/ver4a/onc-kde:42-upstream docker://registry.uncontrol.me/ver4a/onc-kde:42-upstream

Essentially if an image is copied using skopeo from containers-storage transport to docker transport, at least annotations are lost podman-container-tools/skopeo#2686 (not caused by missing annotations) and bootc upgrade/switch to such an image using docker transport fails.

root@desktop:~# bootc switch registry.uncontrol.me/ver4a/onc-kde:42-upstream
layers already present: 0; layers needed: 65 (3.1 GB)
Fetched layers: 0 B in 0 seconds (0 B/s)
error: Switching: Pulling: Importing: Unencapsulating base: serialization error: invalid value: integer `-1`, expected u64

If such an image is copied to containers-storage beforehand, bootc switch --transport containers-storage works.

root@desktop:~# bootc switch --transport containers-storage registry.uncontrol.me/ver4a/onc-kde:42-upstream
layers already present: 4; layers needed: 61 (2.6 GB)
Fetched layers: 2.43 GiB in 47 seconds (53.23 MiB/s)
Deploying: done (20 seconds)
Pruned images: 0 (layers: 0, objsize: 1.6 GB)

This behavior changed between 1.4.0 (which works) and 1.5.1, I've also tested latest builds after 1.7.1 was released and they had the same behavior as 1.5.1.

error: Switching: Pulling: Importing: Unencapsulating base: serialization error: invalid value: integer `-1`, expected u64

[cgwalters]

Essentially if an image is copied using skopeo from containers-storage transport to docker transport, [...]
invalid value: integer -1,

Humm...this may relate to youki-dev/oci-spec-rs#203

I wonder if docker is explicitly putting -1 in there in some cases?

[ggiguash]

This problem also surfaced in the configuration involving a Quay mirror registry. The images are copied into the mirror using skopeo docker://<src_image_ref> docker://<dst_image_ref> command.

Then, bootc image switch commands fail with the above mentioned errors (manifests itself in kickstart)

[   39.355642] anaconda[2704]: Installing.
[   40.358941] anaconda[2704]: The installation was stopped due to an error which occurred while running in non-interactive cmdline mode. Since there cannot be any questions in cmdline mode, edit your kickstart file and retry installation.
[   40.359010] anaconda[2704]: The exact error message is:
[   40.359030] anaconda[2704]: Non interactive installation failed: The command 'ostree container image deploy --sysroot=/mnt/sysimage --image=192.168.122.1:5000/microshift/cos9-bootc-source --transport=registry' exited with the code 1:
[   40.359048] anaconda[2704]: error: Performing deployment: Importing: Unencapsulating base: serialization error: invalid value: integer `-1`, expected u64
[   40.359065] anaconda[2704]: .
[   40.359081] anaconda[2704]: The installer will now terminate.

This is a showstopper for MicroShift CI because it uses mirror registry unconditionally.

[cgwalters]

Briefly investigating the chain looks like

• Change the type of size to be u64 youki-dev/oci-spec-rs#203
• Released in https://github.com/youki-dev/oci-spec-rs/releases/tag/v0.7.0
• Port to oci-spec 0.7 containers-image-proxy-rs#67 updates to that tag
• Released in bootc-dev/containers-image-proxy-rs@38f1e8b
• Updated bootc in https://github.com/bootc-dev/bootc/commits/e4468f73d4b5a90aaa79238714fd7a597c3f45bd
• Released in https://github.com/bootc-dev/bootc/releases/tag/v1.7.1

Hmm oh but wait we also used it in ostree-ext pre-merge into this repo, and that bump came in ostreedev/ostree-rs-ext@e4468f7
which then came into bootc via a8737d9 ?

Anyways I guess the timeline doesn't really matter and we just need to fix this.

[cgwalters]

I wonder if docker is explicitly putting -1 in there in some cases?

Of course I got confused by the naming of the docker:// transport - this has nothing to do with the docker software AFAICS and is just about registries, and must relate to specific registry versions/software. It looks like both reporters here are using a private quay.

[cgwalters]

I tried installing https://github.com/quay/mirror-registry but am not immediately reproducing this problem. What version of the mirror registry are people using?

Although navigating to https://registry.uncontrol.me/ I see it claims to be "Quay v3.15.1" and I am confused as that version number does not appear in https://github.com/quay/quay/releases/

[cgwalters]

Can someone with a reproducer environment get the raw output of the manifest from the target registry? i.e.

skopeo inspect --raw docker://192.168.122.1:5000/microshift/cos9-bootc-source

If that's a manifest listed image you'll need to get the --raw output of a per-arch manifest.

[ggiguash]

Can someone with a reproducer environment get the raw output of the manifest from the target registry?

The output of the following command is attached

skopeo inspect --authfile ./_output/test-images/mirror-registry/config/microshift_auth.json --raw docker://192.168.122.1:5000/microshift/centos-bootc/centos-bootc:stream9

skopeo_inspect.txt

[ver4a]

Although navigating to https://registry.uncontrol.me/ I see it claims to be "Quay v3.15.1" and I am confused as that version number does not appear in https://github.com/quay/quay/releases/

It's a bit further down, because there are multiple maintained releases, but it's this one https://github.com/quay/quay/releases/tag/v3.15.1.

You can reproduce it by switching to/inspecting this image registry.uncontrol.me/ver4a/onc-kde:42-cantpull

[cgwalters]

bootc-dev/containers-image-proxy-rs#97