Skip to content

ci.md

Latest commit

 

History

History
58 lines (35 loc) · 3.31 KB

ci.md

File metadata and controls

58 lines (35 loc) · 3.31 KB

CI for InstructLab Training

Before running any testing locally, ensure you have run pip install -r requirements-dev.txt in your environment.

Unit tests

Unit tests are designed to test specific training components or features in isolation. Generally, new code should be adding or modifying unit tests.

All unit tests currently live in the tests/unit directory and are run with pytest via tox.

To run the unit tests, you can run tox -e py3-unit.

In CI, the tests are run with Python 3.11 - 3.13 on Ubuntu and MacOS runners - you can see the details in the unit tests workflow file.

Smoke tests

Some smoke tests currently live in the tests/smoke directory and are run with pytest via tox.

There is also a Shell-based smoke test script that can be found at tests/smoketest.sh.

To run the smoke tests, you can run tox -e py3-smoke

In CI, the smoke tests are run with Python 3.11 on CentOS runners - you can see the details in the smoke workflow file.

End-to-end (E2E) tests

InstructLab training has several end-to-end jobs that run to ensure compatibility with the InstructLab Core project. You can see details about the types of jobs being run in the matrix below.

For more details about the E2E scripts themselves, see the InstructLab Core documentation.

Current E2E Jobs

Name T-Shirt Size Runner Host Instance Type OS GPU Type Script Flags Runs when? Discord reporting?
e2e-nvidia-l40s-x4.yml Large AWS g6e.12xlarge CentOS Stream 9 4 x NVIDIA L40S w/ 48 GB VRAM (192 GB) e2e-ci.sh l Manually by Maintainers, Automatically against main branch at 4PM UTC Yes

Discord reporting

Some E2E jobs send their results to the channel #e2e-ci-results via the Son of Jeeves bot in Discord. You can see which jobs currently have reporting via the "Current E2E Jobs" table above.

In Discord, we use actions/actions-status-discord and the built-in channel webhooks feature.

Triggering an E2E job via GitHub Web UI

For the E2E jobs that can be launched manually, they take an input field that specifies the PR number or git branch to run them against. If you run them against a PR, they will automatically post a comment to the PR when the tests begin and end so it's easier for those involved in the PR to follow the results.

  1. Visit the Actions tab.
  2. Click on one of the E2E workflows on the left side of the page.
  3. Click on the Run workflow button on the right side of the page.
  4. Enter a branch name or a PR number in the input field.
  5. Click the green Run workflow button.

Note

Only users with "Write" permissions to the repo can run CI jobs manually