Skip to content

Commit 7693016

Browse files
thanodnlhanefi
thanodnl
and
hanefi
authored
Move CI images to GHCR (#157)
The main motivation of this patch is to move the images to the github container registry. But as quite often the case, the images were not building, thus some changes needed to be made to get the images to build. - Python construct package is bumped. There was a dependency failing to install, which succeeds on a newer version. This change is made by changing the version on the citus repo, and re-export the requirements.txt - pin the `postgresql-$PG_MAJOR` version to the same pgdg version. For pg16.2 new packages have been created. This happens once in a while and to counter that we already pin most of the postgres ecosystem packages to a specific pgdg version. Somehow `postgresql-...` wasn't part of the pinning, causing some errors during image builds. The scripts used the old syntax of multi line run blocks, requiring a lot of escaping and chaining of lines with the trailing `\` character. Newer versions of docker allow an easier to maintain and readable syntax via ```Dockerfile RUN <<'EOF' # first comment will be shown during build as 'name' of the task ... EOF ``` Given I already needed to change the body of many of the RUN blocks I took the liberty to refactor this at the same time. --------- Co-authored-by: Hanefi Onaldi <Hanefi.Onaldi@microsoft.com>
1 parent 390dab3 commit 7693016

9 files changed

Lines changed: 224 additions & 128 deletions

File tree

‎.github/workflows/build-test-images.yml‎

Lines changed: 15 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -38,11 +38,12 @@ jobs:
3838
- name: Checkout repository
3939
uses: actions/checkout@v2
4040

41-
- name: Login to Docker Hub
42-
uses: docker/login-action@v1
41+
- name: 'Login to GitHub Container Registry'
42+
uses: docker/login-action@v3
4343
with:
44-
username: ${{ secrets.DOCKERHUB_USER_NAME }}
45-
password: ${{ secrets.DOCKERHUB_PASSWORD }}
44+
registry: ghcr.io
45+
username: ${{ github.actor }}
46+
password: ${{ secrets.GITHUB_TOKEN }}
4647

4748
# If current branch is not master,build and publish dev image
4849
- name: Build & Push all dev images
@@ -66,11 +67,12 @@ jobs:
6667
- name: Checkout repository
6768
uses: actions/checkout@v2
6869

69-
- name: Login to Docker Hub
70-
uses: docker/login-action@v1
70+
- name: 'Login to GitHub Container Registry'
71+
uses: docker/login-action@v3
7172
with:
72-
username: ${{ secrets.DOCKERHUB_USER_NAME }}
73-
password: ${{ secrets.DOCKERHUB_PASSWORD }}
73+
registry: ghcr.io
74+
username: ${{ github.actor }}
75+
password: ${{ secrets.GITHUB_TOKEN }}
7476

7577
# If current branch is not master,build and publish dev image
7678
- name: Build & Push all dev images
@@ -90,11 +92,12 @@ jobs:
9092
- name: Checkout repository
9193
uses: actions/checkout@v2
9294

93-
- name: Login to Docker Hub
94-
uses: docker/login-action@v1
95+
- name: 'Login to GitHub Container Registry'
96+
uses: docker/login-action@v3
9597
with:
96-
username: ${{ secrets.DOCKERHUB_USER_NAME }}
97-
password: ${{ secrets.DOCKERHUB_PASSWORD }}
98+
registry: ghcr.io
99+
username: ${{ github.actor }}
100+
password: ${{ secrets.GITHUB_TOKEN }}
98101

99102
- name: Build & Push all live images
100103
run: |

‎circleci/images/Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# config variable
2-
DOCKER_REPO ?= citus
2+
DOCKER_REPO ?= ghcr.io/citusdata
33
SHA_SUFFIX := $(shell git rev-parse --short HEAD)
44

55
# auto generated variables

‎circleci/images/citusupgradetester/Dockerfile‎

Lines changed: 36 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,12 @@ FROM python:3.9.7-slim-bullseye AS base
44
# add unpriviliged user for tests
55
RUN useradd -ms /bin/bash circleci
66

7-
RUN apt-get update \
7+
RUN <<'EOF'
88
# install dependencies
9-
&& apt-get install -y --no-install-recommends \
9+
set -eux
10+
apt-get update
11+
12+
apt-get install -y --no-install-recommends \
1013
apt-transport-https \
1114
autoconf \
1215
build-essential \
@@ -34,7 +37,8 @@ RUN apt-get update \
3437
perl \
3538

3639
# clear apt cache
37-
&& rm -rf /var/lib/apt/lists/*
40+
rm -rf /var/lib/apt/lists/*
41+
EOF
3842

3943
# make special locales available
4044
COPY locale.gen /etc/locale.gen
@@ -43,28 +47,37 @@ RUN locale-gen
4347
# Add codeclimate
4448
RUN curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > /usr/local/bin/cc-test-reporter && chmod +x /usr/local/bin/cc-test-reporter
4549

46-
# add postgres signing key
47-
RUN curl -sf https://www.postgresql.org/media/keys/ACCC4CF8.asc | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add -
48-
4950
ARG PG_VERSION
5051
ARG PG_MAJOR
5152
ENV PG_VERSION=$PG_VERSION
5253
ENV PG_MAJOR=$PG_MAJOR
5354

54-
# add postgres repository
55-
RUN echo "deb https://apt.postgresql.org/pub/repos/apt bullseye-pgdg main" >> /etc/apt/sources.list.d/postgresql.list \
56-
&& echo "deb https://apt-archive.postgresql.org/pub/repos/apt bullseye-pgdg-archive main" >> /etc/apt/sources.list.d/postgresql.list \
57-
&& apt-get update \
55+
56+
RUN <<'EOF'
57+
# install postgres ecosystem for pg version: $PG_VERSION
58+
set -eux
59+
60+
# install key and repositories
61+
curl -sf https://www.postgresql.org/media/keys/ACCC4CF8.asc | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add -
62+
echo "deb https://apt.postgresql.org/pub/repos/apt bullseye-pgdg main" >> /etc/apt/sources.list.d/postgresql.list
63+
echo "deb https://apt-archive.postgresql.org/pub/repos/apt bullseye-pgdg-archive main" >> /etc/apt/sources.list.d/postgresql.list
64+
65+
apt-get update
66+
5867
# infer the pgdgversion of postgres based on the $PG_VERSION
59-
&& pgdg_version=$(apt list -a postgresql-server-dev-${PG_MAJOR} 2>/dev/null | grep "${PG_VERSION}" | awk '{print $2}' | head -n1 ) \
60-
&& apt-get install -y --no-install-recommends --allow-downgrades \
68+
pgdg_version=$(apt list -a postgresql-server-dev-${PG_MAJOR} 2>/dev/null | grep "${PG_VERSION}" | awk '{print $2}' | head -n1 )
69+
70+
apt-get install -y --no-install-recommends --allow-downgrades \
6171
libpq-dev=${pgdg_version} \
6272
libpq5=${pgdg_version} \
73+
postgresql-${PG_MAJOR}=${pgdg_version} \
6374
postgresql-client-${PG_MAJOR}=${pgdg_version} \
6475
postgresql-${PG_MAJOR}-dbgsym=${pgdg_version} \
6576
postgresql-server-dev-${PG_MAJOR}=${pgdg_version} \
77+
6678
# clear apt cache
67-
&& rm -rf /var/lib/apt/lists/*
79+
rm -rf /var/lib/apt/lists/*
80+
EOF
6881

6982
# add postgress to the path
7083
ENV PATH=/usr/lib/postgresql/$PG_MAJOR/bin/:$PATH
@@ -75,11 +88,18 @@ RUN mkdir -p /var/run/postgresql && chown -R postgres:postgres /var/run/postgres
7588
##############################################################################
7689
FROM base AS builder
7790

78-
RUN apt update \
79-
&& apt install -y \
91+
RUN <<'EOF'
92+
# install dependencies
93+
set -eux
94+
apt update
95+
96+
apt install -y \
8097
git \
8198
libkrb5-dev \
82-
&& rm -rf /var/lib/apt/lists/*
99+
100+
# clear apt cache
101+
rm -rf /var/lib/apt/lists/*
102+
EOF
83103

84104
ARG CITUS_VERSIONS
85105
ENV CITUS_VERSIONS=$CITUS_VERSIONS
@@ -98,8 +118,6 @@ RUN chown -R circleci /usr/lib/postgresql/ /usr/include/postgresql/ /usr/share/p
98118
COPY ./files/etc/requirements.txt /tmp/etc/
99119
RUN pip3 install -Ir /tmp/etc/requirements.txt
100120

101-
WORKDIR /home/circleci
102-
103121
ARG CITUS_VERSIONS
104122
ENV CITUS_VERSIONS=$CITUS_VERSIONS
105123

‎circleci/images/extbuilder/Dockerfile‎

Lines changed: 27 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,12 @@ FROM buildpack-deps:bullseye
33
# add unpriviliged user for tests
44
RUN useradd -ms /bin/bash circleci
55

6-
RUN apt-get update \
6+
RUN <<'EOF'
77
# install dependencies
8-
&& apt-get install -y --no-install-recommends \
8+
set -eux
9+
apt-get update
10+
11+
apt-get install -y --no-install-recommends \
912
apt-transport-https \
1013
autoconf \
1114
build-essential \
@@ -29,31 +32,41 @@ RUN apt-get update \
2932
locales \
3033
make \
3134
perl \
32-
# clear apt cache
33-
&& rm -rf /var/lib/apt/lists/*
3435

35-
# add postgres signing key
36-
RUN curl -sf https://www.postgresql.org/media/keys/ACCC4CF8.asc | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add -
36+
# clear apt cache
37+
rm -rf /var/lib/apt/lists/*
38+
EOF
3739

3840
ARG PG_VERSION
3941
ARG PG_MAJOR
4042
ENV PG_VERSION=$PG_VERSION
4143
ENV PG_MAJOR=$PG_MAJOR
4244

43-
# add postgres repository
44-
RUN echo "deb https://apt.postgresql.org/pub/repos/apt bullseye-pgdg main" >> /etc/apt/sources.list.d/postgresql.list \
45-
&& echo "deb https://apt-archive.postgresql.org/pub/repos/apt bullseye-pgdg-archive main" >> /etc/apt/sources.list.d/postgresql.list \
46-
&& apt-get update \
45+
RUN <<'EOF'
46+
# install postgres ecosystem for pg version: $PG_VERSION
47+
set -eux
48+
49+
# install key and repositories
50+
curl -sf https://www.postgresql.org/media/keys/ACCC4CF8.asc | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add -
51+
echo "deb https://apt.postgresql.org/pub/repos/apt bullseye-pgdg main" >> /etc/apt/sources.list.d/postgresql.list
52+
echo "deb https://apt-archive.postgresql.org/pub/repos/apt bullseye-pgdg-archive main" >> /etc/apt/sources.list.d/postgresql.list
53+
54+
apt-get update
55+
4756
# infer the pgdgversion of postgres based on the $PG_VERSION
48-
&& pgdg_version=$(apt list -a postgresql-server-dev-${PG_MAJOR} 2>/dev/null | grep "${PG_VERSION}" | awk '{print $2}' | head -n1 ) \
49-
&& apt-get install -y --no-install-recommends --allow-downgrades \
57+
pgdg_version=$(apt list -a postgresql-server-dev-${PG_MAJOR} 2>/dev/null | grep "${PG_VERSION}" | awk '{print $2}' | head -n1 )
58+
59+
apt-get install -y --no-install-recommends --allow-downgrades \
5060
libpq-dev=${pgdg_version} \
5161
libpq5=${pgdg_version} \
62+
postgresql-${PG_MAJOR}=${pgdg_version} \
5263
postgresql-client-${PG_MAJOR}=${pgdg_version} \
5364
postgresql-${PG_MAJOR}-dbgsym=${pgdg_version} \
54-
postgresql-server-dev-${PG_MAJOR}=${pgdg_version} \
65+
postgresql-server-dev-${PG_MAJOR}=${pgdg_version}
66+
5567
# clear apt cache
56-
&& rm -rf /var/lib/apt/lists/*
68+
rm -rf /var/lib/apt/lists/*
69+
EOF
5770

5871
USER circleci
5972
WORKDIR /home/circleci

‎circleci/images/exttester/Dockerfile‎

Lines changed: 49 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,13 @@
22
# we use this builder to build the isolation tester from a specific version
33
FROM buildpack-deps:bullseye AS dev-tools-builder
44

5-
RUN apt-get update \
5+
RUN <<'EOF'
66
# install dependencies
7-
&& apt-get install -y --no-install-recommends \
7+
set -eux
8+
9+
apt-get update
10+
11+
apt-get install -y --no-install-recommends \
812
apt-transport-https \
913
autoconf \
1014
build-essential \
@@ -28,8 +32,10 @@ RUN apt-get update \
2832
locales \
2933
make \
3034
perl \
35+
3136
# clear apt cache
32-
&& rm -rf /var/lib/apt/lists/*
37+
rm -rf /var/lib/apt/lists/*
38+
EOF
3339

3440
ARG PG_VERSION
3541
ARG PG_VERSION_CLEAN
@@ -43,11 +49,15 @@ RUN tar jxf "postgresql-${PG_VERSION_CLEAN}.tar.bz2"
4349
# apply optional patches that might be required for a successful testsuite
4450
WORKDIR /build/postgresql-${PG_VERSION_CLEAN}/
4551
COPY patches/ patches/
46-
SHELL ["/bin/bash", "-c"]
47-
RUN if [ -d "patches/${PG_VERSION_CLEAN}/" ]; \
48-
then \
49-
git apply patches/${PG_VERSION_CLEAN}/*.patch; \
50-
fi;
52+
RUN <<'EOF'
53+
# apply postgres patches
54+
55+
set -eux
56+
if [ -d "patches/${PG_VERSION_CLEAN}/" ];
57+
then
58+
git apply patches/${PG_VERSION_CLEAN}/*.patch;
59+
fi;
60+
EOF
5161

5262
WORKDIR /build/postgresql-${PG_VERSION_CLEAN}/build
5363
RUN ../configure --prefix /usr/lib/postgresql/${PG_MAJOR}/
@@ -63,9 +73,9 @@ ARG PG_VERSION_CLEAN
6373
ARG PG_MAJOR
6474

6575
COPY --from=dev-tools-builder \
66-
/build/postgresql-${PG_VERSION_CLEAN}/build/src/test/isolation/pg_isolation_regress \
67-
/build/postgresql-${PG_VERSION_CLEAN}/build/src/test/isolation/isolationtester \
68-
build/postgresql-${PG_MAJOR}/build/src/test/isolation/
76+
/build/postgresql-${PG_VERSION_CLEAN}/build/src/test/isolation/pg_isolation_regress \
77+
/build/postgresql-${PG_VERSION_CLEAN}/build/src/test/isolation/isolationtester \
78+
build/postgresql-${PG_MAJOR}/build/src/test/isolation/
6979

7080
# copy regress files in multiple layers as we only need a few
7181
COPY --from=dev-tools-builder /build/postgresql-${PG_VERSION_CLEAN}/build/src/test/regress/regress.so usr/lib/postgresql/${PG_MAJOR}/lib/
@@ -77,9 +87,13 @@ FROM buildpack-deps:bullseye
7787
# add unpriviliged user for tests
7888
RUN useradd -ms /bin/bash circleci
7989

80-
RUN apt-get update \
90+
RUN <<'EOF'
8191
# install dependencies
82-
&& apt-get install -y --no-install-recommends \
92+
set -eux
93+
94+
apt-get update
95+
96+
apt-get install -y --no-install-recommends \
8397
apt-transport-https \
8498
autoconf \
8599
build-essential \
@@ -106,8 +120,10 @@ RUN apt-get update \
106120
locales \
107121
make \
108122
perl \
123+
109124
# clear apt cache
110-
&& rm -rf /var/lib/apt/lists/*
125+
rm -rf /var/lib/apt/lists/*
126+
EOF
111127

112128
# install rust
113129
RUN curl --proto '=https' --tlsv1.2 --silent --show-error https://sh.rustup.rs | sh -s -- -y
@@ -123,31 +139,39 @@ RUN locale-gen
123139
# Add codeclimate
124140
RUN curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > /usr/local/bin/cc-test-reporter && chmod +x /usr/local/bin/cc-test-reporter
125141

126-
# add postgres signing key
127-
RUN curl -sf https://www.postgresql.org/media/keys/ACCC4CF8.asc | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add -
128-
129142
ARG PG_VERSION
130143
ARG PG_MAJOR
131144
ENV PG_VERSION=$PG_VERSION
132145
ENV PG_MAJOR=$PG_MAJOR
133146

134-
# add postgres repository
135-
RUN echo "deb https://apt.postgresql.org/pub/repos/apt bullseye-pgdg main" >> /etc/apt/sources.list.d/postgresql.list \
136-
&& echo "deb https://apt-archive.postgresql.org/pub/repos/apt bullseye-pgdg-archive main" >> /etc/apt/sources.list.d/postgresql.list \
137-
&& apt-get update \
147+
RUN <<'EOF'
148+
# install postgres ecosystem for pg version: $PG_VERSION
149+
set -eux
150+
151+
# install key and repositories
152+
curl -sf https://www.postgresql.org/media/keys/ACCC4CF8.asc | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add -
153+
echo "deb https://apt.postgresql.org/pub/repos/apt bullseye-pgdg main" >> /etc/apt/sources.list.d/postgresql.list
154+
echo "deb https://apt-archive.postgresql.org/pub/repos/apt bullseye-pgdg-archive main" >> /etc/apt/sources.list.d/postgresql.list
155+
156+
apt-get update
157+
138158
# infer the pgdgversion of postgres based on the $PG_VERSION
139-
&& pgdg_version=$(apt list -a postgresql-server-dev-${PG_MAJOR} 2>/dev/null | grep "${PG_VERSION}" | awk '{print $2}' | head -n1 ) \
140-
&& apt-get install -y --no-install-recommends --allow-downgrades \
159+
pgdg_version=$(apt list -a postgresql-server-dev-${PG_MAJOR} 2>/dev/null | grep "${PG_VERSION}" | awk '{print $2}' | head -n1 )
160+
161+
apt-get install -y --no-install-recommends --allow-downgrades \
141162
libdbi-perl \
142163
libdbd-pg-perl \
143164
libpq-dev=${pgdg_version} \
144165
libpq5=${pgdg_version} \
166+
postgresql-${PG_MAJOR}=${pgdg_version} \
145167
postgresql-client-${PG_MAJOR}=${pgdg_version} \
146168
postgresql-${PG_MAJOR}-dbgsym=${pgdg_version} \
147169
postgresql-server-dev-${PG_MAJOR}=${pgdg_version} \
148170
postgresql-${PG_MAJOR}-wal2json \
149-
# clear apt cache
150-
&& rm -rf /var/lib/apt/lists/*
171+
172+
# clear apt cache
173+
rm -rf /var/lib/apt/lists/*
174+
EOF
151175

152176
# add postgress to the path
153177
ENV PATH=/usr/lib/postgresql/$PG_MAJOR/bin/:$PATH

0 commit comments

Comments
 (0)