Add PG13 to our test images (#38)

* Add PG13 to out test images

* Remove unnecessary packages, update readme

Author: SaitTalhaNisanci

README.md

@@ -87,17 +87,24 @@ And for all images follow the instructions on [how to publish a change](#4-How-t
 
 If you want to add support for a major postgres major, like `pg-13` you will need to do the steps that are not limited to:
 
-* Generate `postgresql-server-dev-{pg-major}` package with `debbuilder` image. Make sure to add the package index for `pg-major` in `/etc/apt/sources.list.d/pgdg.list`. You can see find the related part [here](https://github.com/citusdata/the-process/blob/master/circleci/images/debbuilder/files/install-builddeps).
+* Generate `postgresql-server-dev-{pg-major}` package with `debbuilder` image. Make sure to add the package index for `pg-major` in `/etc/apt/sources.list.d/pgdg.list`(See the [pgdg apt repository](https://apt.postgresql.org/pub/repos/apt/dists/)). You can update `pg_latest` variable in [install-builddebs](https://github.com/citusdata/the-process/blob/master/circleci/images/debbuilder/files/install-builddeps).
+
+```bash
+cd debbuilder
+docker build --tag=citus/debbuilder13 . --build-arg PG_MAJORS=13
+```
+
 * The generated package will be in `{container-name}/home/circleci/debs`. One way of getting the package from the docker image is:
   * Run the container with `docker run -it {tag-name} bash`
   * Find the name of your container with `docker ps`
   * Copy the `debs` folder from docker container to your local `docker cp {container-name}:/home/circleci/debs .`
-* Upload the package to [the-process](https://packagecloud.io/citus-bot/the-process). Make sure that you choose `debian stretch` while uploading. In order to upload you can:
+* Upload the package to [the-process](https://packagecloud.io/citus-bot/the-process). Make sure that you choose `debian stretch` while uploading. Use `citus-bot` account there(credentials are in the 1password vault). In order to upload you can:
   * Upload the package file from the UI by clicking to `Upload image`
   * Or you can use the [package cloud cli](https://packagecloud.io/l/cli).
 
 * Add `pg-major` to `extbuilder` in its script so that citus artifacts are generated for that `pg_major` too.
-* Add new `pg-major` package index to `sources.list`:
+* Update `pg_latest`.
+* (This step should already be done if you update `pg_latest`). Add new `pg-major` package index to `sources.list`:
 
 ```bash
 # add pgdg repo to sources
@@ -106,6 +113,11 @@ echo "deb http://apt.postgresql.org/pub/repos/apt/ ${codename}-pgdg main ${PG-MA
     /etc/apt/sources.list.d/postgresql.list
 ```
 
+```bash
+cd extbuilder
+docker build --tag=citus/extbuilder-{pg-major} .
+```
+
 * Build `exttester` with `pg-major`:
 
 ```bash

circleci/images/debbuilder/files/build-pgdebs

@@ -82,6 +82,8 @@ build_deb() {
     r /tmp/install-isolation.mk
     d
   }' src/test/isolation/Makefile
+  echo 'usr/lib/postgresql/*/lib/pgxs/src/test/isolation/*' >> \
+       "debian/postgresql-server-dev-${pg_major}.install"
   tweak+=("Add PostgreSQL's isolation tester to installed files")
 
   sed -i '/^Package: postgresql-server-dev-/,/^Breaks:$/ {/\(clang\|llvm\)/d}' debian/control

circleci/images/debbuilder/files/install-builddeps

@@ -7,6 +7,8 @@ IFS=$'\n\t'
 codename=stretch
 pg_majors=( "${@}" )
 
+pg_latest=13
+
 # explicitly set user/group IDs
 groupadd -r circleci --gid=1729
 useradd -mg circleci --uid=1729 --shell=/bin/bash circleci
@@ -23,17 +25,21 @@ echo "Writing /etc/apt/sources.list.d/pgdg.list..." >&2
 cat > /etc/apt/sources.list.d/pgdg.list <<EOF
 deb     http://deb.debian.org/debian ${codename}-backports                main
 deb-src http://apt.postgresql.org/pub/repos/apt/ ${codename}-pgdg         main
-deb-src http://apt.postgresql.org/pub/repos/apt/ ${codename}-pgdg-testing 12
+deb-src http://apt.postgresql.org/pub/repos/apt/ ${codename}-pgdg-testing ${pg_latest}
 EOF
 
 echo "Running apt-get update..." >&2
 apt-get update
 
+# this line is to bypass password prompts
+echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
 echo "Installing tools for building Debian packages..." >&2
 apt-get install -y --no-install-recommends \
+  apt-utils \
   devscripts \
   fakeroot \
-  lintian
+  lintian 
+
 
 pg_pkgs=( "${pg_majors[@]/#/postgresql-}" )
 apt-get build-dep -y "${pg_pkgs[@]}"

circleci/images/extbuilder/files/bin/build-ext

@@ -5,7 +5,7 @@ set -euxo pipefail
 IFS=$'\n\t'
 
 # supported PostgreSQL releases
-pg_majors=( 11 12 )
+pg_majors=( 11 12 13)
 
 # get codename from release file
 . /etc/os-release

circleci/images/extbuilder/files/sbin/install-extdeps

@@ -11,9 +11,9 @@ if [[ $EUID -ne 0 ]]; then
 fi
 
 # supported PostgreSQL releases
-pg_majors=( 11 12 )
+pg_majors=( 11 12 13 )
 
-pg_latest=12
+pg_latest=13
 
 # get codename from release file
 . /etc/os-release

circleci/images/exttester/files/sbin/install-testdeps

@@ -15,7 +15,7 @@ fi
 codename=${VERSION#*(}
 codename=${codename%)*}
 
-pg_latest=12
+pg_latest=13
 
 # explicitly set user/group IDs
 groupadd -r circleci --gid=1729

circleci/images/failtester/files/sbin/install-testdeps

@@ -15,6 +15,8 @@ fi
 codename=${VERSION#*(}
 codename=${codename%)*}
 
+pg_latest=13
+
 # explicitly set user/group IDs
 groupadd -r circleci --gid=1729
 useradd -mg circleci --uid=1729 --shell=/bin/bash circleci
@@ -35,7 +37,7 @@ APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add -
 
 # add pgdg repo to sources
 echo "Writing /etc/apt/sources.list.d/pgdg.list..." >&2
-echo "deb http://apt.postgresql.org/pub/repos/apt/ ${codename}-pgdg main 12" > \
+echo "deb http://apt.postgresql.org/pub/repos/apt/ ${codename}-pgdg main ${pg_latest}" > \
      /etc/apt/sources.list.d/pgdg.list  
 
 pkg_cloud_url="https://packagecloud.io/install/repositories/citus-bot/the-process/script.deb.sh"

circleci/images/pgupgradetester/files/sbin/install-testdeps

@@ -16,9 +16,9 @@ codename=${VERSION#*(}
 codename=${codename%)*}
 
 # supported PostgreSQL releases
-pg_majors=( 10 11 12 )
+pg_majors=( 11 12 13 )
 
-pg_latest=12
+pg_latest=13
 
 # explicitly set user/group IDs
 groupadd -r circleci --gid=1729
@@ -47,11 +47,6 @@ echo "deb http://apt.postgresql.org/pub/repos/apt/ ${codename}-pgdg main ${pg_la
 pkg_cloud_url="https://packagecloud.io/install/repositories/citus-bot/the-process/script.deb.sh"
 curl -sf "${pkg_cloud_url}" | unique_id=exttester bash
 
-cat > /etc/apt/preferences.d/citusdata_the-process.pref <<EOF
-Package: postgresql-server-dev-*
-Pin: release o=packagecloud.io/citus-bot/the-process
-Pin-Priority: 700
-EOF
 
 echo "Installing tools for testing PostgreSQL extensions..." >&2
 sd_pkgs=( "${pg_majors[@]/#/postgresql-server-dev-}" )