Skip to content

podman auth: export REGISTRY_AUTH_FILE in every podman-invoking shell - #2668

Open
djgalloway wants to merge 1 commit into
mainfrom
podman-auth-same-shell
Open

podman auth: export REGISTRY_AUTH_FILE in every podman-invoking shell#2668
djgalloway wants to merge 1 commit into
mainfrom
podman-auth-same-shell

Conversation

@djgalloway

@djgalloway djgalloway commented Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

Three commits:

  1. Revert the ceph-dev-pipeline: pin podman auth to a persistent authfile #2656 merge (ceph-dev-pipeline --authfile logins + Groovy env.REGISTRY_AUTH_FILE)
  2. Revert aae7f2f (bwc.sh --authfile)
  3. Fix: every sh block that invokes podman or build-with-container.py — builder container stage login/pull/build/push, build stage bwc runs, and bwc_login — exports REGISTRY_AUTH_FILE="$HOME/.config/containers/auth.json" itself. Plain podman login, no flags.

Why

https://tracker.ceph.com/issues/77920 recurred on 2026-07-22 with #2656 deployed: login wrote docker.io creds to $HOME/.config/containers/auth.json, and the FROM docker.io/ubuntu:22.04 pull inside podman build (spawned by build-with-container.py) still pulled anonymously → toomanyrequests.

Debug run on a builder pinned it:

$ env -u XDG_RUNTIME_DIR REGISTRY_AUTH_FILE=$HOME/.config/containers/auth.json \
    podman build --pull --no-cache --log-level=debug -f /tmp/df /tmp 2>&1 | grep -iE 'credential|account='
msg="Found credentials for docker.io/library/ubuntu in credential helper containers-auth.json in file /home/jenkins-build/.config/containers/auth.json"
msg="GET https://auth.docker.io/token?account=dgalloway&..."

podman build's base-image pull honors REGISTRY_AUTH_FILE when it's actually in the process environment — so on 2026-07-22 it wasn't. Hence:

  • --authfile on the login alone is worse than nothing: it steers credentials into a file later pulls never consult.
  • Groovy env threading demonstrably failed to deliver the variable to the bwc subprocess.
  • A same-shell export reaches podman by plain process inheritance and can't be silently dropped. podman login honors it for writes, so login and all later reads use the same persistent file by construction. Repeated verbatim per block on purpose: self-sufficient and greppable.

Also retracts aae7f2f's logind-teardown rationale: a week of auditd on /run/user/<uid>/containers/ showed every auth.json write was podman's own atomic tmp+rename, unbroken inode chain, no external deletions.

Fixes: https://tracker.ceph.com/issues/77920

@djgalloway
djgalloway force-pushed the podman-auth-same-shell branch from a2a69ee to d67f8bc Compare July 28, 2026 21:24
idryomov
idryomov reviewed Jul 31, 2026
View reviewed changes

@idryomov idryomov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest squashing the reverts as a good chunk of their diff is immediately re-added in the "podman auth: share one authfile via same-shell exports" commit.

Comment thread ceph-dev-pipeline/build/Jenkinsfile
def ceph_builder_tag_short = "${env.BRANCH}.${env.DIST}.${env.ARCH}.${env.FLAVOR}"
def ceph_builder_tag = "${env.SHA1[0..6]}.${ceph_builder_tag_short}"
sh """#!/bin/bash -ex
export REGISTRY_AUTH_FILE="\${HOME}/.config/containers/auth.json"

@idryomov idryomov Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a significance to not expanding $HOME here and other places below (except in bwc.sh)?

@djgalloway djgalloway Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I'm understanding your question correctly, the \ is Groovy escaping, not a shell thing

@idryomov idryomov Aug 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't notice that the sh block above is ''' while this one and the ones below are """ which necessitates escaping.

@djgalloway
podman auth: share one authfile via same-shell exports
ebb51dd 
This reverts the two previous attempts at fixing the Docker Hub
rate-limit failures -- commit aae7f2f ("bwc: pin podman auth to a
persistent authfile") and PR #2656 (merge f199616, the
ceph-dev-pipeline equivalent) -- and replaces both. They were built on
wrong assumptions:

- aae7f2f blamed logind tearing down the XDG runtime dir between
  steps. An auditd watch on /run/user/<uid>/containers/ disproved
  that: nothing external ever touches auth.json there; every write is
  podman login's own atomic tmp+rename. Linger is enabled and the
  agent runs as a systemd service with no login sessions, so the
  directory is stable across the whole job.
- PR #2656 added --authfile to the logins plus a Groovy
  env.REGISTRY_AUTH_FILE assignment. Tracker 77920 recurred on
  2026-07-22 with it deployed: login wrote docker.io credentials to
  $HOME/.config/containers/auth.json, and the
  FROM docker.io/ubuntu:22.04 pull inside podman build (spawned by
  build-with-container.py) still ran anonymous and hit
  toomanyrequests.

A debug run on a builder pinned the real mechanism. With the variable
present in podman build's environment, the base image pull uses it:

  $ REGISTRY_AUTH_FILE=$HOME/.config/containers/auth.json \
      podman build --pull --log-level=debug ... 2>&1 | grep -i cred
  "Found credentials for docker.io/library/ubuntu ... in file
   /home/jenkins-build/.config/containers/auth.json"

So the pull authenticates whenever REGISTRY_AUTH_FILE actually reaches
the process, which means on 2026-07-22 it did not. That dictates the
shape of this change:

- No --authfile on logins: pointing only the login at a file steers
  credentials somewhere later pulls never look, which is worse than
  the podman default.
- No Groovy env threading: it demonstrably failed to deliver the
  variable to the bwc subprocess, and it silently depends on stage
  ordering and when{} guards.
- Instead, every sh block that invokes podman or
  build-with-container.py exports REGISTRY_AUTH_FILE itself. A
  same-shell export reaches podman through ordinary process
  inheritance and cannot be lost. podman login honors the variable
  for writes, so login and every later read use the same persistent
  file by construction. The export line is repeated verbatim in each
  block on purpose: each block is self-sufficient and the pattern is
  greppable.

Fixes: https://tracker.ceph.com/issues/77920

Signed-off-by: David Galloway <david.galloway@ibm.com>
@djgalloway
djgalloway force-pushed the podman-auth-same-shell branch from d67f8bc to ebb51dd Compare July 31, 2026 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@idryomov idryomov idryomov left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@djgalloway @idryomov