Skip to content

Commit cc3191e

Browse files
bkeryanbkeryan
authored
Speed up GitHub workflows (#486)
* GitHub workflows: Try using setup-python's Poetry caching * GitHub workflows: Use pipx to install Poetry * GitHub workflows: Don't bother with cache for Publish_NIMS.yml * GitHub workflows: Set cache-dependency-path * GitHub workflows: redo caching * GitHub workflow: Cache example virtualenvs * GitHub workflows: Cache Tox virtualenvs * GitHub workflows: Fix a restore/save mixup * GitHub workflows: Add ids to cache steps * GitHub workflows: Stop testing on macOS * GitHub workflows: Cache example poetry.lock files * GitHub workflows: Add gzip to the path * GitHub workflows: Revert PATH hack I updated the AMI build to add UNIX tools to the path and install zstandard, so actions/cache should work without this hack (which didn't work anyway). * GitHub workflows: Revert to Gr1N/setup-poetry to ensure we use the correct Python versions for testing * GitHub workflows: Change cache key to use "py3.9" instead of just "3.9" * GitHub workflows: Move all/no-extras next to package name in cache key * GitHub workflows: Don't skip poetry install on cache hit `poetry install` is fast when the dependencies are already installed. Also, if the venv was relocated to a different path, it may fix the venv.
1 parent bb8eae7 commit cc3191e

6 files changed

Lines changed: 113 additions & 27 deletions

File tree

‎.github/workflows/Publish_NIMS.yml‎

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -25,22 +25,20 @@ jobs:
2525
name: Update API reference docs and Publish NIMS Package to PyPI
2626
runs-on : ubuntu-latest
2727
steps:
28-
- uses: actions/checkout@v4
29-
- uses: actions/setup-python@v4
28+
- name: Check out repo
29+
uses: actions/checkout@v4
30+
- name: Set up Python
31+
uses: actions/setup-python@v4
3032
with:
3133
python-version: ${{ env.PYTHON_VERSION }}
32-
- uses: Gr1N/setup-poetry@v8
34+
- name: Set up Poetry
35+
uses: Gr1N/setup-poetry@v8
3336
with:
3437
poetry-version: ${{ env.POETRY_VERSION }}
3538

3639
- name: Check for lock changes
3740
run: poetry lock --check
3841

39-
- uses: actions/cache@v3
40-
with:
41-
path: ~/.cache/pypoetry/virtualenvs
42-
key: ${{ runner.os }}-poetry-${{ hashFiles('poetry.lock') }}
43-
4442
# If the tag is 0.1.0, this will set the version of NIMS package to 0.1.0
4543
- name: Store version from Tag
4644
id: vars

‎.github/workflows/check_examples.yml‎

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -23,12 +23,30 @@ jobs:
2323
uses: Gr1N/setup-poetry@v8
2424
with:
2525
poetry-version: ${{ env.POETRY_VERSION }}
26-
- name: Cache Poetry virtualenv
26+
# Updating poetry.lock for all of the examples takes over 6 minutes, so it's worth caching.
27+
- name: Cache poetry.lock
2728
uses: actions/cache@v3
29+
id: cache-poetry-lock
2830
with:
29-
path: ~/.cache/pypoetry/virtualenvs
30-
key: ${{ runner.os }}-poetry-${{ hashFiles('poetry.lock') }}
31-
# Install each example's dependencies so mypy can see their types.
31+
path: 'examples/**/poetry.lock'
32+
# Include the main project's poetry.lock in the hash to detect upstream dependency updates.
33+
key: examples-poetry-lock-${{ runner.os }}-py${{ env.PYTHON_VERSION }}-${{ hashFiles('examples/**/pyproject.toml', 'poetry.lock') }}
34+
- name: Lock examples
35+
if: steps.cache-poetry-lock.outputs.cache-hit != 'true'
36+
run: |
37+
for example in examples/*/; do
38+
echo "::group::$example"
39+
pushd $example
40+
poetry lock
41+
popd
42+
echo "::endgroup::"
43+
done
44+
- name: Cache virtualenvs
45+
uses: actions/cache@v3
46+
id: cache-venv
47+
with:
48+
path: 'examples/**/.venv'
49+
key: examples-venv-${{ runner.os }}-py${{ env.PYTHON_VERSION }}-${{ hashFiles('examples/**/poetry.lock') }}
3250
- name: Install examples
3351
run: |
3452
for example in examples/*/; do

‎.github/workflows/check_nimg.yml‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,11 +29,12 @@ jobs:
2929
poetry-version: ${{ env.POETRY_VERSION }}
3030
- name: Check for lock changes (ni-measurementlink-generator)
3131
run: poetry lock --check
32-
- name: Cache Poetry virtualenv
32+
- name: Cache virtualenv (ni-measurementlink-generator)
3333
uses: actions/cache@v3
34+
id: cache
3435
with:
35-
path: ~/.cache/pypoetry/virtualenvs
36-
key: ${{ runner.os }}-poetry-${{ hashFiles('poetry.lock') }}
36+
path: ni_measurementlink_generator/.venv
37+
key: ni-measurementlink-generator-${{ runner.os }}-py${{ env.PYTHON_VERSION }}-${{ hashFiles('poetry.lock') }}
3738
- name: Install ni-measurementlink-generator
3839
run: poetry install -v
3940
- name: Lint ni-measurementlink-generator

‎.github/workflows/check_nims.yml‎

Lines changed: 28 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -25,13 +25,22 @@ jobs:
2525
poetry-version: ${{ env.POETRY_VERSION }}
2626
- name: Check for lock changes (ni-measurementlink-service)
2727
run: poetry lock --check
28-
- name: Cache Poetry virtualenv
29-
uses: actions/cache@v3
28+
29+
# ni-measurementlink-service, all extras
30+
- name: Restore cached virtualenv (ni-measurementlink-service, all extras)
31+
uses: actions/cache/restore@v3
32+
id: restore-nims-all-extras
3033
with:
31-
path: ~/.cache/pypoetry/virtualenvs
32-
key: ${{ runner.os }}-poetry-${{ hashFiles('poetry.lock') }}
34+
path: .venv
35+
key: ni-measurementlink-service-all-extras-${{ runner.os }}-py${{ env.PYTHON_VERSION }}-${{ hashFiles('poetry.lock') }}
3336
- name: Install ni-measurementlink-service (all extras)
3437
run: poetry install -v --all-extras
38+
- name: Save cached virtualenv (ni-measurementlink-service, all extras)
39+
uses: actions/cache/save@v3
40+
if: steps.restore-nims-all-extras.outputs.cache-hit != 'true'
41+
with:
42+
path: .venv
43+
key: ${{ steps.restore-nims-all-extras.outputs.cache-primary-key }}
3544
- name: Lint ni-measurementlink-service
3645
run: poetry run ni-python-styleguide lint
3746
- name: Mypy static analysis (ni-measurementlink-service, Linux)
@@ -42,12 +51,26 @@ jobs:
4251
run: poetry run mypy tests
4352
- name: Mypy static analysis (tests, Windows)
4453
run: poetry run mypy tests --platform win32
54+
55+
# ni-measurementlink-service, all extras, docs
56+
- name: Restore cached virtualenv (ni-measurementlink-service, all extras, docs)
57+
uses: actions/cache/restore@v3
58+
id: restore-nims-all-extras-docs
59+
with:
60+
path: .venv
61+
key: ni-measurementlink-service-all-extras-docs-${{ runner.os }}-py${{ env.PYTHON_VERSION }}-${{ hashFiles('poetry.lock') }}
4562
- name: Install ni-measurementlink-service (all extras, docs)
4663
run: poetry install -v --all-extras --with docs
64+
- name: Save cached virtualenv (ni-measurementlink-service, all extras, docs)
65+
uses: actions/cache/save@v3
66+
if: steps.restore-nims-all-extras-docs.outputs.cache-hit != 'true'
67+
with:
68+
path: .venv
69+
key: ${{ steps.restore-nims-all-extras-docs.outputs.cache-primary-key }}
4770
- name: Build docs and check for errors/warnings
4871
run: |
4972
rm -rf docs
5073
mkdir -p docs
5174
poetry run sphinx-build _docs_source docs -b html -W --keep-going
5275
- name: Revert docs
53-
run: git clean -dfx docs/ && git restore docs/
76+
run: git clean -dfx docs/ && git restore docs/

‎.github/workflows/run_system_tests.yml‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,13 +22,21 @@ jobs:
2222
steps:
2323
- name: Check out repo
2424
uses: actions/checkout@v4
25+
- name: Cache virtualenvs
26+
uses: actions/cache@v3
27+
id: cache
28+
with:
29+
path: |
30+
.venv
31+
.tox
32+
key: run-system-tests-${{ runner.os }}-${{ matrix.configuration }}-${{ hashFiles('poetry.lock') }}
2533
- name: Install dependencies
26-
run: poetry install
34+
run: poetry install -v
2735
- name: Run system tests
2836
run: poetry run tox
2937
- name: Upload test results
3038
uses: actions/upload-artifact@v3
3139
with:
3240
name: test_results
3341
path: test_results/*.xml
34-
if: always()
42+
if: always()

‎.github/workflows/run_unit_tests.yml‎

Lines changed: 43 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ jobs:
1313
runs-on: ${{ matrix.os }}
1414
strategy:
1515
matrix:
16-
os: [macos-latest, windows-latest, ubuntu-latest]
16+
os: [windows-latest, ubuntu-latest]
1717
python-version: [3.8, 3.9, '3.10', 3.11, 3.12]
1818
# Fail-fast skews the pass/fail ratio and seems to make pytest produce
1919
# incomplete JUnit XML results.
@@ -29,25 +29,63 @@ jobs:
2929
uses: Gr1N/setup-poetry@v8
3030
with:
3131
poetry-version: ${{ env.POETRY_VERSION }}
32-
- name: Cache Poetry virtualenv
33-
uses: actions/cache@v3
32+
33+
# ni-measurementlink-service, no extras
34+
- name: Restore cached virtualenv (ni-measurementlink-service, no extras)
35+
uses: actions/cache/restore@v3
36+
id: restore-nims-no-extras
3437
with:
35-
path: ~/.cache/pypoetry/virtualenvs
36-
key: ${{ runner.os }}-poetry-${{ hashFiles('poetry.lock') }}
38+
path: .venv
39+
key: ni-measurementlink-service-no-extras-${{ runner.os }}-py${{ matrix.python-version }}-${{ hashFiles('poetry.lock') }}
3740
- name: Install ni-measurementlink-service (no extras)
3841
run: poetry install -v
42+
- name: Save cached virtualenv (ni-measurementlink-service, no extras)
43+
uses: actions/cache/save@v3
44+
if: steps.restore-nims-no-extras.outputs.cache-hit != 'true'
45+
with:
46+
path: .venv
47+
key: ${{ steps.restore-nims-no-extras.outputs.cache-primary-key }}
3948
- name: Run unit tests and code coverage (ni-measurementlink-service, no extras)
4049
run: poetry run pytest ./tests/unit -v --cov=ni_measurementlink_service --junitxml=test_results/nims-${{ matrix.os }}-py${{ matrix.python-version}}-no-extras.xml
50+
51+
# ni-measurementlink-service, all extras
52+
- name: Restore cached virtualenv (ni-measurementlink-service, all extras)
53+
uses: actions/cache/restore@v3
54+
id: restore-nims-all-extras
55+
with:
56+
path: .venv
57+
key: ni-measurementlink-service-all-extras-${{ runner.os }}-py${{ matrix.python-version }}-${{ hashFiles('poetry.lock') }}
4158
- name: Install ni-measurementlink-service (all extras)
4259
run: poetry install -v --all-extras
60+
- name: Save cached ni-measurementlink-service virtualenv (all extras)
61+
uses: actions/cache/save@v3
62+
if: steps.restore-nims-all-extras.outputs.cache-hit != 'true'
63+
with:
64+
path: .venv
65+
key: ${{ steps.restore-nims-all-extras.outputs.cache-primary-key }}
4366
- name: Run unit tests and code coverage (ni-measurementlink-service, all extras)
4467
run: poetry run pytest ./tests/unit -v --cov=ni_measurementlink_service --junitxml=test_results/nims-${{ matrix.os }}-py${{ matrix.python-version}}-all-extras.xml
68+
69+
# ni-measurementlink-generator
70+
- name: Restore cached virtualenv (ni-measurementlink-generator)
71+
uses: actions/cache/restore@v3
72+
id: restore-nimg
73+
with:
74+
path: ni_measurementlink_generator/.venv
75+
key: ni-measurementlink-generator-${{ runner.os }}-py${{ matrix.python-version }}-${{ hashFiles('poetry.lock') }}
4576
- name: Install ni-measurementlink-generator
4677
run: poetry install -v
4778
working-directory: ./ni_measurementlink_generator
79+
- name: Save cached virtualenv (ni-measurementlink-generator)
80+
uses: actions/cache/save@v3
81+
if: steps.restore-nimg.outputs.cache-hit != 'true'
82+
with:
83+
path: ni_measurementlink_generator/.venv
84+
key: ${{ steps.restore-nimg.outputs.cache-primary-key }}
4885
- name: Run tests and code coverage (ni-measurementlink-generator)
4986
run: poetry run pytest -v --cov=ni_measurementlink_generator --junitxml=test_results/nimg-${{ matrix.os }}-py${{ matrix.python-version}}.xml
5087
working-directory: ./ni_measurementlink_generator
88+
5189
- name: Upload test results
5290
uses: actions/upload-artifact@v3
5391
with:

0 commit comments

Comments
 (0)