Skip to content

Commit 83e0e1d

Browse files
scottgigante-immunaiscottgigante-immunai
authored
Fix benchmark trigger (#400)
* try using nf-core/tower-action * shortcut * fix cancel * faster * compute_env * try a different ID * use name not ID * try curl * comment cleanup * syntax * POST * set run_name * fix name * fix * typo * add process_results workflow * install deps * just fetch the data from aws * cp * argv * fix workflow * cp quiet * full ci * typo * fix logic * return s3fs and cli * pass github_pat * fix cancel * Revert "fix cancel" This reverts commit fd54e7d. * more logic fixes * use pytestmark * tighter diff * need to retest everything is tests are modified * temporary speed up * use a different here-keyword * more speedup * more speedup * remove indent * summary * indent * separate test and full benchmark jobs * fix skip * typo * remove matrix * pull on main * don't send github pat in params * output to stderr * temporarily remove most tasks * put everything back in * simplify logic * better logic * redirect stdout * use 1.2
1 parent 8ce4467 commit 83e0e1d

8 files changed

Lines changed: 116 additions & 47 deletions

File tree

‎.github/workflows/run_tests.yml‎

Lines changed: 72 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ jobs:
2121
build_images:
2222

2323
needs: cancel_previous_runs
24-
runs-on: ${{ matrix.config.os }}
24+
runs-on: ubuntu-18.04
2525
if: |
2626
!endsWith(github.event.head_commit.message, '# ci skip') &&
2727
(
@@ -35,12 +35,6 @@ jobs:
3535
env:
3636
BRANCH_NAME: "auto_update_docker_${{ github.run_number }}"
3737

38-
strategy:
39-
fail-fast: false
40-
matrix:
41-
config:
42-
- {name: 'current', os: ubuntu-18.04, python: '3.8' }
43-
4438
steps:
4539
- uses: actions/checkout@v2
4640
with:
@@ -49,7 +43,7 @@ jobs:
4943
- name: Set up Python
5044
uses: actions/setup-python@v2
5145
with:
52-
python-version: ${{ matrix.config.python }}
46+
python-version: "3.8"
5347

5448
- name: Install package & dependencies
5549
run: |
@@ -126,9 +120,10 @@ jobs:
126120
(
127121
startsWith(github.ref, 'refs/heads') ||
128122
startsWith(github.ref, 'refs/tags') ||
129-
github.event.pull_request.draft == false &&
130-
!startsWith(github.ref, 'refs/heads/test_docker')
131-
)
123+
github.event.pull_request.draft == false
124+
) &&
125+
!startsWith(github.ref, 'refs/heads/test_docker') &&
126+
!startsWith(github.ref, 'refs/heads/test_benchmark')
132127
133128
steps:
134129

@@ -173,6 +168,13 @@ jobs:
173168
install_renv("docker/openproblems-r-base/r_requirements.txt")
174169
shell: Rscript {0}
175170

171+
- name: Pull Docker images
172+
if: "startsWith(github.ref, 'refs/heads/main') && github.repository == 'openproblems-bio/openproblems'"
173+
run: |
174+
cd workflow
175+
snakemake -j $(nproc) docker_pull
176+
cd ..
177+
176178
- name: Update Docker images
177179
if: "!(startsWith(github.ref, 'refs/heads/main') && github.repository == 'openproblems-bio/openproblems')"
178180
run: |
@@ -195,20 +197,15 @@ jobs:
195197
path: results
196198

197199

198-
run_benchmark:
200+
run_test_benchmark:
199201
needs: run_tester
200-
runs-on: ${{ matrix.config.os }}
202+
runs-on: ubuntu-18.04
201203
if: >-
202204
always() &&
203205
!endsWith(github.event.head_commit.message, '# ci skip') &&
204206
github.event_name == 'push' &&
205-
needs.run_tester.result == 'success'
206-
207-
strategy:
208-
fail-fast: false
209-
matrix:
210-
config:
211-
- {name: 'current', os: ubuntu-18.04, python: '3.8', java_version: 15 }
207+
needs.run_tester.result == 'success' ||
208+
!startsWith(github.ref, 'refs/heads/test_benchmark')
212209
213210
steps:
214211

@@ -217,25 +214,22 @@ jobs:
217214
fetch-depth: 1000
218215

219216
- name: Install system dependencies
220-
if: runner.os == 'Linux'
221217
run: |
222218
sudo apt-get update -qq || (sudo rm /etc/apt/sources.list.d/* && sudo apt-get update -yqq)
223219
sudo apt-get install -qy libhdf5-dev pandoc gfortran libblas-dev liblapack-dev libedit-dev llvm-dev
224220
225221
- name: Set up Python
226222
uses: actions/setup-python@v2
227223
with:
228-
python-version: ${{ matrix.config.python }}
224+
python-version: "3.8"
229225

230226
- name: Set up Java ${{ matrix.java_version }}
231227
uses: actions/setup-java@v1
232228
with:
233-
java-version: ${{matrix.config.java_version}}
229+
java-version: 15
234230
architecture: x64
235231

236232
- name: Set up environment
237-
env:
238-
TOWER_ACCESS_TOKEN: ${{ secrets.TOWER_ACCESS_KEY }}
239233
run: |
240234
SCRIPTS_PATH=$(python3 -c 'import os, sysconfig; print(sysconfig.get_path("scripts",f"{os.name}_user"))')
241235
echo "PATH=${SCRIPTS_PATH}:${PATH}" >> $GITHUB_ENV
@@ -249,8 +243,6 @@ jobs:
249243
else
250244
echo "BRANCH=`echo ${{ github.repository }} | awk '{print $1}' FS=/ | head -c 6`-`echo ${{ github.ref }} | sed 's:refs/[a-z]*/::' | sed 's:[/_]:-:g'| head -c 33`"; \
251245
fi >> $GITHUB_ENV
252-
# don't use tower if the env var is not set
253-
echo "WITH_TOWER=`if [ $TOWER_ACCESS_TOKEN ]; then echo "-with-tower"; else echo; fi`" >> $GITHUB_ENV
254246
255247
- name: Cache Python packages
256248
uses: actions/cache@v2
@@ -306,14 +298,20 @@ jobs:
306298
openproblems-cli --test-hash
307299
308300
- name: Pull Docker images
309-
if: "(startsWith(github.ref, 'refs/heads/main') && github.repository == 'openproblems-bio/openproblems')"
301+
if: |
302+
startsWith(github.ref, 'refs/heads/main') &&
303+
github.repository == 'openproblems-bio/openproblems'
310304
run: |
311305
cd workflow
312306
snakemake -j $(nproc) docker_pull
313307
cd ..
314308
315309
- name: Build Docker images
316-
if: "!(startsWith(github.ref, 'refs/heads/main') && github.repository == 'openproblems-bio/openproblems')"
310+
if: |
311+
!(
312+
startsWith(github.ref, 'refs/heads/main') &&
313+
github.repository == 'openproblems-bio/openproblems'
314+
)
317315
run: |
318316
cd workflow
319317
snakemake -j $(nproc) docker
@@ -338,13 +336,15 @@ jobs:
338336
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
339337
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
340338
TOWER_ACCESS_TOKEN: ${{ secrets.TOWER_ACCESS_KEY }}
341-
AWS_DEFAULT_REGION: us-west-2
342339
TOWER_WORKSPACE_ID: 53907369739130
340+
AWS_DEFAULT_REGION: us-west-2
343341
NXF_DEFAULT_DSL: 1
344342
run: |
345343
RUN_NAME="$(echo "$BRANCH" | sed "s/[^a-z]//g")_$(git rev-parse --short HEAD)_${GITHUB_RUN_ATTEMPT}"
346344
cd /mnt/openproblems-nextflow/cwd/${BRANCH}
347-
nextflow run $WITH_TOWER \
345+
nextflow run \
346+
-revision v1.2 \
347+
-with-tower \
348348
-ansi-log false -resume \
349349
-profile aws,test \
350350
-work-dir "/mnt/openproblems-nextflow/work/${BRANCH}" \
@@ -364,7 +364,11 @@ jobs:
364364
path: /mnt/openproblems-nextflow/cwd/${{ env.BRANCH }}/.nextflow.log
365365

366366
- name: AWS ECR cleanup
367-
if: success() && !startsWith(github.ref, 'refs/heads/main')
367+
if: |
368+
!(
369+
startsWith(github.ref, 'refs/heads/main') &&
370+
github.repository == 'openproblems-bio/openproblems'
371+
)
368372
env:
369373
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
370374
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
@@ -375,7 +379,10 @@ jobs:
375379
done
376380
377381
- name: AWS S3 cleanup
378-
if: success() || startsWith(github.ref, 'refs/heads/main') || startsWith(github.ref, 'refs/tags')
382+
if: |
383+
success() ||
384+
startsWith(github.ref, 'refs/heads/main') ||
385+
startsWith(github.ref, 'refs/tags')
379386
env:
380387
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
381388
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
@@ -385,21 +392,42 @@ jobs:
385392
aws s3 rm --recursive "s3://openproblems-nextflow/bucket/${{ env.BRANCH }}"
386393
aws s3 rm --recursive "s3://openproblems-nextflow/cwd/${{ env.BRANCH }}"
387394
395+
run_full_benchmark:
396+
needs: run_test_benchmark
397+
runs-on: ubuntu-latest
398+
if: >-
399+
always() &&
400+
(needs.run_test_benchmark.result == 'success' || needs.run_test_benchmark.result == 'skipped') &&
401+
!endsWith(github.event.head_commit.message, '# ci skip') &&
402+
github.event_name == 'push' &&
403+
(
404+
startsWith(github.ref, 'refs/tags') ||
405+
startsWith(github.ref, 'refs/heads/test_benchmark')
406+
)
407+
408+
steps:
409+
388410
- name: Run full benchmark
389-
if: |
390-
(startsWith(github.ref, 'refs/heads/test_benchmark') || startsWith(github.ref, 'refs/tags')) &&
391-
github.repository == 'openproblems-bio/openproblems'
411+
env:
412+
TOWER_WATCH_URL: https://tower.nf/orgs/openproblems-bio/workspaces/openproblems-bio/watch
413+
TOWER_WORKSPACE_ID: 53907369739130
392414
run: |
393415
generate_parameters()
394416
{
395-
cat <<EOF
417+
cat <<EOF
396418
{
397-
"branch": "main",
398-
"github_pat": "${{ secrets.GH_WORKFLOW_PAT }}"
419+
"params" : {
420+
"branch": "main"
421+
}
399422
}
400-
EOF
423+
EOF
401424
}
402-
curl -H "Content-Type: application/json" \
403-
-H "Authorization: Bearer ${{ secrets.TOWER_ACCESS_KEY }}" \
404-
https://api.tower.nf/actions/2kxpi4Q0zNCy6wPExD8yD5/launch?workspaceId=53907369739130 \
405-
--data "$(generate_parameters)"
425+
WORKFLOW_ID=$(
426+
curl -H "Content-Type: application/json" \
427+
-H "Authorization: Bearer ${{ secrets.TOWER_ACCESS_KEY }}" \
428+
https://api.tower.nf/actions/2kxpi4Q0zNCy6wPExD8yD5/launch?workspaceId=${TOWER_WORKSPACE_ID} \
429+
--data "$(generate_parameters)" | \
430+
tee /dev/stderr | \
431+
sed 's/.*"\([a-zA-Z0-9]*\)".*/\1/'
432+
)
433+
echo "Full benchmark running at ${TOWER_WATCH_URL}/${WORKFLOW_ID}" >> $GITHUB_STEP_SUMMARY

‎openproblems/api/main.py‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,13 @@ def _main(args=None):
3838

3939

4040
def main(args=None, do_print=True):
41-
"""Run the command-line interface."""
42-
output = _main(args)
41+
"""Run the command-line interface.
42+
43+
Since printing the output to stdout is important here,
44+
we redirect all other stdout to stderr.
45+
"""
46+
with utils.RedirectStdout():
47+
output = _main(args)
4348
if do_print:
4449
utils.print_output(output)
4550
return 0

‎openproblems/api/utils.py‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,23 @@
11
import openproblems
2+
import sys
3+
4+
5+
class RedirectStdout(object):
6+
def __init__(self):
7+
self._stdout = sys.stdout
8+
self._stderr = sys.stderr
9+
10+
def _flush(self):
11+
self._stdout.flush()
12+
self._stderr.flush()
13+
14+
def __enter__(self):
15+
self._flush()
16+
sys.stdout = self._stderr
17+
18+
def __exit__(self, exc_type, exc_value, traceback):
19+
self._flush()
20+
sys.stdout = self._stdout
221

322

423
def module_to_str(module):

‎test/test__load_data.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,15 @@
11
import openproblems
22
import parameterized
3+
import pytest
34
import utils.docker
45
import utils.git
56
import utils.name
67
import utils.warnings
78

89
utils.warnings.ignore_warnings()
10+
pytestmark = pytest.mark.skipif(
11+
len(utils.git.list_modified_tasks()) == 0, reason="No tasks have been modified"
12+
)
913

1014

1115
@parameterized.parameterized.expand(

‎test/test_datasets.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@
1313
import utils.warnings
1414

1515
utils.warnings.ignore_warnings()
16+
pytestmark = pytest.mark.skipif(
17+
len(utils.git.list_modified_tasks()) == 0, reason="No tasks have been modified"
18+
)
1619

1720

1821
def _assert_not_bytes(X):

‎test/test_methods.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,15 @@
11
import openproblems
22
import parameterized
3+
import pytest
34
import utils.docker
45
import utils.git
56
import utils.name
67
import utils.warnings
78

89
utils.warnings.ignore_warnings()
10+
pytestmark = pytest.mark.skipif(
11+
len(utils.git.list_modified_tasks()) == 0, reason="No tasks have been modified"
12+
)
913

1014

1115
@parameterized.parameterized.expand(

‎test/test_metrics.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,14 @@
11
import openproblems
22
import parameterized
3+
import pytest
34
import utils.git
45
import utils.name
56
import utils.warnings
67

78
utils.warnings.ignore_warnings()
9+
pytestmark = pytest.mark.skipif(
10+
len(utils.git.list_modified_tasks()) == 0, reason="No tasks have been modified"
11+
)
812

913

1014
@parameterized.parameterized.expand(

‎test/utils/git.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,9 @@ def core_modified():
7070
We exclude all task directories as well as any md files and the website.
7171
"""
7272
task_exclusions = [f":^{task_dir(task)}" for task in openproblems.TASKS]
73-
return git_has_diff([".", ":^*.md", ":^website"] + task_exclusions)
73+
diff_target = ["./openproblems", "./docker", "./test", ":^*.md", ":^website"]
74+
diff_target += task_exclusions
75+
return git_has_diff(diff_target)
7476

7577

7678
def git_rev_parse(branch):

0 commit comments

Comments
 (0)