Skip to content

ceph-dev-pipeline: Add a README - #2672

Open
djgalloway wants to merge 1 commit into
mainfrom
wip-ceph-dev-pipeline-readme
Open

ceph-dev-pipeline: Add a README#2672
djgalloway wants to merge 1 commit into
mainfrom
wip-ceph-dev-pipeline-readme

Conversation

@djgalloway

Copy link
Copy Markdown
Contributor

Documents how ceph-dev-pipeline actually works. No functional changes — this adds ceph-dev-pipeline/README.md only.

What it covers

  • Stage flow — source distribution → matrix (node / checkout ceph-build / copy artifacts / existence checks / builder container / build / upload / container / registry upload), and the fact that matrix cells run on CentOS 9 or Noble hosts regardless of the target distro.
  • Where scripts come from — the two independent sources in a cell's workspace, which is the part that trips people up:
    • ./scripts/ from the ceph/ceph-build clone at $CEPH_BUILD_BRANCH (upload/shaman/pulp/podman glue)

    • ./dist/ceph/ extracted from the source tarball produced by the setup job (build-with-container.py, Dockerfile.build, install-deps.sh, debian/, container/)

      So everything running inside the build container comes from the branch being built, not from a checkout.

  • The builder container's base OS image — not pinned anywhere in ceph-build. Dockerfile.build is FROM $DISTRO, where DISTRO is a --build-arg set from build-with-container.py's DefaultImage map (centos9 → quay.io/centos/centos:stream9, rocky10 → docker.io/rockylinux/rockylinux:10, noble → docker.io/ubuntu:24.04, etc.). Includes the tag scheme, the /ceph bind mount, and why REGISTRY_AUTH_FILE is set.
  • Builder image vs. runtime Ceph dev image — the two are easy to conflate; the latter is built by scripts/build_containercontainer/build.sh with its own hardcoded FROM_IMAGE map.
  • Why SETUP_JOB existsbuild-with-container.py landed in ceph.git in Aug 2024 and wasn't backported to reef until Feb 2025 (ceph#61683), so reef branches had to go through ceph-dev-new-setup + ceph-dev-new instead. ceph-trigger-build routes between the two on the CEPH-BUILD-JOB trailer. ceph-source-dist became the pipeline's default setup job in 6e1f737 with ceph-dev-new-setup kept as an option; the two are interchangeable because both archive the same dist/** layout the copy-artifacts stage filters on.
  • Key parameters table.

Notes for reviewers

Content was derived by reading the Jenkinsfile, the JJB definition, scripts/, and ceph.git's build-with-container.py / Dockerfile.build / container/Containerfile — worth a sanity check on the base-image table and the SETUP_JOB history in particular.

Document how the job works: the stage flow, the two distinct sources of
scripts in a matrix cell's workspace (the ceph-build clone providing
./scripts/, and the ceph source tarball from the setup job providing
./dist/ceph/), and where the builder container's base OS image actually
comes from -- build-with-container.py's DefaultImage map applied to
Dockerfile.build, not anything pinned in ceph-build.

Also clarifies the frequently-conflated builder image (ceph-build) vs.
the runtime Ceph dev image built by scripts/build_container, and records
why SETUP_JOB exists: reef predated the build-with-container.py backport
and had to be built by ceph-dev-new-setup/ceph-dev-new instead.

Signed-off-by: David Galloway <david.galloway@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant