Skip to content

Commit fd1ed6f

Browse files
treld-w-moore
trel
and
d-w-moore
committed
[_697] Github action for running PRC tests
Co-authored-by: d-w-moore <dmoore@renci.org>
1 parent 76ce0e6 commit fd1ed6f

16 files changed

Lines changed: 437 additions & 7 deletions

File tree

‎.github/workflows/run-the-tests.yml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
name: run-the-tests
2+
3+
on: [push, pull_request]
4+
5+
jobs:
6+
tests:
7+
timeout-minutes: 20
8+
9+
name: ${{ matrix.python }}
10+
runs-on: ubuntu-latest
11+
defaults:
12+
run:
13+
working-directory: ./docker-testing
14+
strategy:
15+
matrix:
16+
python: ['3.9','3.13']
17+
18+
steps:
19+
- name: Checkout
20+
uses: actions/checkout@v4
21+
22+
- name: Start containers
23+
run: ./start_containers.sh "${{ matrix.python }}"
24+
25+
- name: run test
26+
run: |
27+
while :; do
28+
client_container=$(docker ps --format "{{.Names}}"|grep python.client)
29+
[ -n "$client_container" ] && break
30+
sleep 1
31+
done
32+
echo "client_container = [$client_container]"
33+
docker exec "${client_container}" /repo_root/docker-testing/run_tests.sh
34+
35+
- name: Stop containers
36+
if: always()
37+
run: docker compose -f harness-docker-compose.yml down

‎docker-testing/README.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
The file `$REPO/.github/workflows/run-the-tests.yml`
2+
(where `$REPO` is the /path/to/local/python-irodsclient repository)
3+
contains commands for starting the server and client containers and running the PRC
4+
suite in response to a push or pull-request.
5+
6+
The tests suite can also be run on any workstation with "docker compose" installed:
7+
8+
1. cd into top level of $REPO
9+
10+
2. run:
11+
```
12+
./docker-testing/start_containers.sh 3.6
13+
```
14+
This builds and runs the docker images. "3.6" is the version of python desired.
15+
16+
3. run:
17+
```
18+
docker exec <name-of-python-client-container> /repo_root/docker-testing/run_tests.sh
19+
```
20+
(Note: `/repo_root` is an actual literal path, internal to the container.)
21+
You'll see the test output displayed on the console. At completion, xmlrunner outputs are in /tmp.
22+
23+
4. use `docker logs -f` with the provider instance name to tail the irods server log output
24+
25+
DEBUGGING
26+
---------
27+
We can also to run a specific test that we specify by name:
28+
29+
```
30+
$ docker exec -it <name-of-python-client-container> /repo_root/docker_testing/run_tests.sh irods.test.<module>.<class>.<method>
31+
```
32+
33+
Optionally we can also enter the PDB command-line debugger at a place of our choosing in the source code, by stopping on a breakpoint,
34+
and then stepping through code.
35+
36+
The breakpoint can be placed by adding the line
37+
38+
```
39+
import pdb;pdb.set_trace()
40+
```
41+
42+
immediately before the source line in the test code at which we wish to enter the debugger.

‎docker-testing/harness-docker-compose.yml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
version: '3'
2+
3+
services:
4+
irods-catalog:
5+
build:
6+
context: irods_catalog
7+
# 5432 is exposed by default and can conflict with other postgres containers.
8+
# When the metalnx-db service is no longer needed, this stanza can be removed.
9+
ports:
10+
- "5430:5432"
11+
environment:
12+
- POSTGRES_PASSWORD=testpassword
13+
14+
python-client:
15+
build:
16+
context: python_client
17+
args:
18+
- python_version
19+
command:
20+
tail -f /dev/null
21+
volumes:
22+
- ${repo_external}:/repo_root:ro
23+
- /tmp/irods-client-share.py-${python_version}-${parent_pid}:/irods_shared
24+
25+
irods-catalog-provider:
26+
volumes:
27+
- /tmp/irods-client-share.py-${python_version}-${parent_pid}:/irods_shared
28+
build:
29+
context: irods_catalog_provider
30+
shm_size: 500mb
31+
# healthcheck:
32+
# test: ["CMD", "su", "-", "irods", "-c", "ils || exit 1"]
33+
# interval: 10s
34+
# timeout: 10s
35+
# retries: 3
36+
# start_period: 20s
37+
# start_interval: 2s
38+
ports:
39+
- "1247:1247"
40+
depends_on:
41+
- irods-catalog
42+

‎docker-testing/iinit.py‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
#!/usr/bin/env python
2+
3+
from getpass import getpass
4+
from irods.password_obfuscation import encode
5+
import json
6+
import os
7+
import sys
8+
from os import chmod
9+
from os.path import expanduser,exists,join
10+
from getopt import getopt
11+
12+
13+
home_env_path = expanduser('~/.irods')
14+
env_file_path = join(home_env_path,'irods_environment.json')
15+
auth_file_path = join(home_env_path,'.irodsA')
16+
17+
18+
def do_iinit(host, port, user, zone, password):
19+
if not exists(home_env_path):
20+
os.makedirs(home_env_path)
21+
else:
22+
raise RuntimeError('~/.irods already exists')
23+
24+
with open(env_file_path,'w') as env_file:
25+
json.dump ( { "irods_host": host,
26+
"irods_port": int(port),
27+
"irods_user_name": user,
28+
"irods_zone_name": zone }, env_file, indent=4)
29+
with open(auth_file_path,'w') as auth_file:
30+
auth_file.write(encode(password))
31+
chmod (auth_file_path,0o600)
32+
33+
34+
def get_kv_pairs_from_cmdline(*args):
35+
arglist = list(args)
36+
while arglist:
37+
k = arglist.pop(0)
38+
v = arglist.pop(0)
39+
yield k,v
40+
41+
42+
if __name__ == '__main__':
43+
import sys
44+
args = sys.argv[1:]
45+
dct = {k:v for k,v in get_kv_pairs_from_cmdline(*args)}
46+
do_iinit(**dct)

‎docker-testing/irods_catalog/Dockerfile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
FROM postgres:12
2+
3+
COPY init-user-db.sh /docker-entrypoint-initdb.d/init-user-db.sh

‎docker-testing/irods_catalog/init-user-db.sh‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/bin/bash
2+
3+
# Adapted from "Initialization script" in documentation for official Postgres dockerhub:
4+
# https://hub.docker.com/_/postgres/
5+
set -e
6+
7+
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
8+
CREATE DATABASE "ICAT";
9+
CREATE USER irods WITH PASSWORD 'testpassword';
10+
GRANT ALL PRIVILEGES ON DATABASE "ICAT" to irods;
11+
EOSQL

‎docker-testing/irods_catalog_provider/Dockerfile‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
FROM ubuntu:20.04
2+
3+
ENV DEBIAN_FRONTEND=noninteractive
4+
5+
RUN apt-get update && \
6+
apt-get install -y \
7+
apt-transport-https \
8+
gnupg \
9+
wget \
10+
&& \
11+
apt-get clean && \
12+
rm -rf /var/lib/apt/lists/* /tmp/*
13+
14+
RUN wget -qO - https://packages.irods.org/irods-signing-key.asc | apt-key add - && \
15+
echo "deb [arch=amd64] https://packages.irods.org/apt/ focal main" | tee /etc/apt/sources.list.d/renci-irods.list
16+
17+
RUN apt-get update && \
18+
apt-get install -y \
19+
libcurl4-gnutls-dev \
20+
jq \
21+
python3 \
22+
python3-distro \
23+
python3-jsonschema \
24+
python3-pip \
25+
python3-psutil \
26+
python3-requests \
27+
rsyslog \
28+
unixodbc \
29+
gawk \
30+
&& \
31+
apt-get clean && \
32+
rm -rf /var/lib/apt/lists/* /tmp/*
33+
34+
ARG irods_version=4.3.1
35+
ARG irods_package_version_suffix=-0~focal
36+
ARG irods_package_version=${irods_version}${irods_package_version_suffix}
37+
ARG irods_resource_plugin_version=${irods_version}.0${irods_package_version_suffix}
38+
39+
RUN apt-get update && \
40+
apt-get install -y \
41+
irods-database-plugin-postgres=${irods_package_version} \
42+
irods-runtime=${irods_package_version} \
43+
irods-server=${irods_package_version} \
44+
irods-icommands=${irods_package_version} \
45+
irods-resource-plugin-s3=${irods_resource_plugin_version} \
46+
&& \
47+
apt-get clean && \
48+
rm -rf /var/lib/apt/lists/* /tmp/*
49+
50+
COPY setup-${irods_version}.input /
51+
RUN mv /setup-${irods_version}.input /irods_setup.input
52+
53+
WORKDIR /
54+
COPY entrypoint.sh .
55+
COPY send_oneshot .
56+
RUN chmod u+x ./entrypoint.sh
57+
RUN chmod u+x ./send_oneshot
58+
ENTRYPOINT ["./entrypoint.sh"]

‎docker-testing/irods_catalog_provider/entrypoint.sh‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
#! /bin/bash -e
2+
3+
catalog_db_hostname=irods-catalog
4+
5+
echo "Waiting for iRODS catalog database to be ready"
6+
7+
until pg_isready -h ${catalog_db_hostname} -d ICAT -U irods -q
8+
do
9+
sleep 1
10+
done
11+
12+
echo "iRODS catalog database is ready"
13+
14+
setup_input_file=/irods_setup.input
15+
16+
if [ -e "${setup_input_file}" ]; then
17+
echo "Running iRODS setup"
18+
python3 /var/lib/irods/scripts/setup_irods.py < "${setup_input_file}"
19+
rm /irods_setup.input
20+
fi
21+
22+
ORIG_SERVER_CONFIG=/etc/irods/server_config.json
23+
MOD_SERVER_CONFIG=/tmp/server_config.json.$$
24+
25+
#TODO ensure this is done for 4.3+ only. 4.2 doesn't have this server config key
26+
{
27+
[ -f ~/provider-address.do_not_remove ] || {
28+
jq <$ORIG_SERVER_CONFIG >$MOD_SERVER_CONFIG \
29+
'.host_resolution.host_entries += [
30+
{
31+
"address_type": "local",
32+
"addresses": [
33+
"irods-catalog-provider",
34+
"'$(hostname)'"
35+
]
36+
}
37+
]' && \
38+
cat <$MOD_SERVER_CONFIG >$ORIG_SERVER_CONFIG && \
39+
touch ~/provider-address.do_not_remove
40+
}
41+
} || { echo >&2 "Error modifying $ORIG_SERVER_CONFIG"; exit 1; }
42+
43+
echo "Starting server"
44+
45+
# After successful launch of server (per ils success), signal the client container we are ready
46+
{
47+
# wait until server is up
48+
while :; do
49+
su - irods -c ils >/dev/null 2>&1 && break
50+
#echo "** waiting on server before send_oneshot" |tee -a /tmp/debug.dan
51+
sleep 1
52+
done
53+
chown -R irods:irods /irods_shared
54+
echo "**** got this far - about to execute send_oneshot" |tee -a /tmp/debug.dan
55+
env PORT=8888 "$(dirname "$0")"/send_oneshot
56+
} &
57+
58+
cd /usr/sbin
59+
su irods -c 'bash -c "./irodsServer -u"'
60+

‎docker-testing/irods_catalog_provider/send_oneshot‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/usr/bin/gawk -f
2+
BEGIN {
3+
SERVER = "/inet/tcp/"ENVIRON["PORT"]"/0/0"
4+
print ARGV[1] " - " strftime() |& SERVER
5+
close(SERVER)
6+
}

‎docker-testing/irods_catalog_provider/setup-4.3.1.input‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
2+
3+
4+
5+
irods-catalog
6+
5432
7+
ICAT
8+
irods
9+
y
10+
testpassword
11+
12+
y
13+
demoResc
14+
15+
tempZone
16+
1247
17+
20000
18+
20199
19+
1248
20+
21+
rods
22+
y
23+
TEMPORARY_ZONE_KEY
24+
32_byte_server_negotiation_key__
25+
32_byte_server_control_plane_key
26+
rods
27+
28+

0 commit comments

Comments
 (0)