Skip to content

CI: publish test results as artifacts and GitHub Actions step summaries - #1348

Open
ACSimon33 wants to merge 5 commits into
Reference-LAPACK:masterfrom
ACSimon33:test_summary_gh_actions
Open

CI: publish test results as artifacts and GitHub Actions step summaries#1348
ACSimon33 wants to merge 5 commits into
Reference-LAPACK:masterfrom
ACSimon33:test_summary_gh_actions

Conversation

@ACSimon33

Copy link
Copy Markdown
Contributor

Summary

  • lapack_testing.py gains a --markdown PATH option (alongside the existing --junit-xml) that writes a GitHub-flavored Markdown report of the test results.
  • The two CMake test jobs (test-install-release, test-extended-api-only) now upload TESTING/testing_results.txt and lapack_testing_junit.xml as a workflow artifact and render the Markdown report on each job's summary page ($GITHUB_STEP_SUMMARY), with a link to the artifact.

GitHub Actions has no native JUnit ingestion (unlike GitLab, which the --junit-xml option targets), so today the only way to inspect a CI test failure is to scroll through the raw ctest log, and the detailed testing_results.txt produced by every run is discarded when the runner is torn down. With this change each test job shows an at-a-glance verdict, the summary tables, and every failing test set directly on the run page — and keeps the full output for two weeks, which is needed most on exactly the runs that failed.

Copilot AI review requested due to automatic review settings August 2, 2026 15:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request improves CI test observability in GitHub Actions by generating a GitHub-flavored Markdown summary from LAPACK test outputs and preserving raw test artifacts for later inspection.

Changes:

  • Add a --markdown PATH option to lapack_testing.py to emit a GitHub Actions-friendly Markdown report (including collapsible failure details with size caps).
  • Upload TESTING/testing_results.txt (and intended JUnit XML) as workflow artifacts from the CMake test jobs.
  • Append the generated Markdown report to $GITHUB_STEP_SUMMARY, linking to the uploaded artifact.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
lapack_testing.py Adds Markdown report generation and supporting helpers (formatting, truncation/limits, failure rendering).
.github/workflows/cmake.yml Uploads test artifacts and writes a per-job GitHub Actions step summary using the new Markdown report.
Suppressed comments (1)

.github/workflows/cmake.yml:211

  • Same issue as the other job: after cd build, python3 lapack_testing.py won’t find the script (it lives at the repo root), so summary.md/reports may not be produced. This job also uploads build/lapack_testing_junit.xml but doesn’t generate it unless --junit-xml is passed.
        cd build 2>/dev/null || exit 0
        python3 lapack_testing.py -s -d TESTING --merge-apis --markdown summary.md || true
        if [ -f summary.md ]; then

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/cmake.yml
Comment thread lapack_testing.py
ACSimon33 and others added 5 commits August 2, 2026 18:13
Summary tables first, then one entry per failing output file with
its notable lines collapsed; sized in UTF-8 bytes to stay under
GitHub's 1 MiB step-summary limit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
testing_results.txt and the JUnit XML are kept for 14 days even when
the tests fail; the step summary links to the artifact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A nonzero error count whose percentage would display as 0.00% is
shown as 0.01% instead, so it cannot read as a zero error rate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Shares error_percent with the Markdown report, including the 0.01%
floor for nonzero counts; the error columns are rebalanced by one
character so (100.00%) still fits under the rule.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ACSimon33
ACSimon33 force-pushed the test_summary_gh_actions branch from f2e304f to 0a70a8c Compare August 2, 2026 17:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants