ReFrame 4.10.0

Release Notes

New features

• New option --warn-as-error for treating warnings as errors (#3671)
• New configuration option slurm_envvar_whitelist to white list Slurm environment variables during job submission (#3670)
• Remove unnecessary Slurm polling when retrieving job pending reason (#3642)
• Allow git sparse checkouts and passing arbitrary git options when cloning a repo from sourcesdir (#3629)
• Modernize ReFrame installation and remove support for Python < 3.10 (#3651)
• Allow combining explicit system partitions and features/extras in valid_systems (#3613)
• Support multi-cluster operation in Slurm backends (#3639)
• Support custom indexed and external references (#3630)
• Add support for dynamically setting the authorization header in the httpjson log handler (#3624)
• Add support for running the unit tests offline (#3626)
• Support multiple aggregations and statistics at once when querying performance data (#3585)
• Use more flexible build options in CMake backend (#3614)
• Remove support for the legacy environment modules version 3 (#3596)
• Add new pbspro scheduler backend (#3584)
• Treat nodes in MAINTENANCE state as available for flexible allocations on Slurm MAINT-flagged reservations (#3573)

Enhancements

• Add a note in docs about --ci-generate not working with test fixtures (#3673)
• Add note in docs about the scope of target_systems (#3674)
• Fix typo in README regarding reframe executable (#3666)

Bug fixes

• Treat properly Slurm constraints with . (#3667)
• Generate perflog record for XFAIL and XPASS tests (#3656)
• Do not hard code the value type of a RegressionTestDict (#3660)
• Treat . properly in external references keys (#3658)

Other

• Update ReFrame tutorial containers (#3661)
• Update .readthedocs.yaml to use uv for building the docs (#3657)

ReFrame 4.9.3

Release Notes

Enhancements

• Update docs on parameter packs (#3646)

Bug fixes

• Fix default datefmt for the perflog handlers (#3644)

ReFrame 4.9.2

Release Notes

Enhancements

• Update documentation on builtins usage (#3634)

Bug fixes

• Fix ReFrame crash when trying to retrieve source code context during errors and treat Unicode errors gracefully (#3632)

ReFrame 4.9.1

Release Notes

Bug fixes

• Do not crash if regex in hostnames is invalid (#3616)
• Honor the systems.partitions.sched_options.max_sacct_failures configuration option for Slurm backends (#3615)
• Fix multiple test retries for tests with fixtures (#3612)
• Fix AttributeError crash in ssh backend (#3611)
• Properly treat preempted jobs in Slurm backend (#3610)
• Fix typo in RegressionMixin deprecation message (#3607)
• Fix typo in error message (#3589)

ReFrame 4.9.0

Release Notes

New features

• Updates and improvements to the poll control mechanism (#3575)
• Deprecate RegressionMixin in favor of RegressionTestPlugin (#3581)
• Replace perflog_compat with perflog_multiline in logging configuration (#3578)
• Log also results from sanity-only tests through the perflog handlers (#3516)
• Support usecs (%f format) in datefmt for logging (#3570)
• Support OR-ing node states in flexible node allocation (#3569)
• Allow the -P option to redefine existing parameters (#3554)
• Facilitate local imports from configuration files (#3548)
• Add new --table-format-delim option (#3547)
• Add new --param-values-delim option (#3538)
• Add support for marking tests as expected failures (#3481)
• Enable launchers to forward environment variables (#3464)

Enhancements

• Print the performance reference tuple while running only if defined (#3517)
• Document the builtin configuration (#3558)
• Result database performance improvements (#3546)
• Add some notes and tips in the documentation of pipeline hooks (#3540)
• Add link to FOSDEM'25 presentation (#3539)
• Skip tests if their dependencies fail (#3500)
• Update Slack invite link and add a note about expiration (#3499)

Bug fixes

• Treat negative exit codes in PBS scheduler (#3571)
• Fix progress numbering for XFAIL/XPASS tests (#3529)
• Fix failure report crash in case of expected failures (#3527)

Test library

• Fix ROCm build (#3574)

Other

• Update codecov badge (#3506)
• Enable scheduler testing Github workflows on push (#3505)
• Re-enable codecov in the project (#3220)
• Restore deleted Dockerfiles (#3502)

ReFrame 4.8.4

Release Notes

Enhancements

• Documentation enhancements (#3552)
  • Add link to FOSDEM'25 presentation
  • Add some notes and tips in the docs of pipeline hooks

Bug fixes

• Fix resource leaks in osext.run_command() and in _ProcFuture (#3556)
• Do not wipe the output dir of a test if run with --dry-run (#3553)

ReFrame 4.8.3

Release Notes

Enhancements

• Show more test attributes with --describe (#3531)

Bug fixes

• Use proper locking when accessing the SQLite DB (#3533)
• Introduce optional file locking for appending to perflogs (#3523)
• Treat = properly when parameterizing tests with --parameterize (#3530)
• Properly treat forceful termination due to SIGTERM and SIGHUP (#3534)
• Disable file logging when emitting DB records (#3520)

ReFrame 4.8.2

Release Notes

Enhancements

• Improve programming error messages (#3489)

Bug fixes

• Properly cleanup build jobs on abort (#3512)
• Fix race condition in local scheduler during timeout and properly reap done processes (#3511)
• Make -n option syntax consistent across test filtering and results storage queries (#3508)
• Return Job.jobid as string always (#3493)
• Include (again) partition name in test failure info (#3487)

Other

• Bump requests package version (#3501)

ReFrame 4.8.1

Release Notes

Enhancements

• Use the --use-pep517 option in bootstrap.sh (#3478)
• Use ClusterShell to manipulate node lists instead of our own code (#3470)
• Show up front if result storage is enabled (#3472)
• Fix typo in versionadded annotation (#3473)

Bug fixes

• Use always | to join unique values in column aggregations except for --table-format=pretty (#3471)
• Unset all SBATCH_* environment variables before submitting a job (#3474)
• Use display_name when printing the test's name in FAILURE INFO (#3468)

ReFrame 4.8.0

Release Notes

New features

• Support list assignment from JSON using the -S option (#3449)
• Support discrete testcase filters when comparing performance (#3438)
• Do not emit the -lselect option if num_tasks=None (#3452)
• Allow changing the default mount location of stagedir inside containers (#3442)
• Support selecting A/B column variants in performance comparisons (#3431)
• Support reference values of zero (#3434)
• Allow scoping session-based DB queries by time (#3403)
• Store test sub-result per performance variable (#3414)
• Add support for configure directory in CMake builds (#3395)
• Enable YAML configuration files (#3370)
• Introduce %(hostname)s log format placeholder (#3346)

Enhancements

• Document the requirement of file lock support for enabling results storage (#3457)
• Add a note about results storage and analytics in README (#3453)
• Use correct OMP_PROC_BIND arguments in the tutorial examples (#3435)
• List parameterized tests in their instantiation order (#3440)
• Add warning about combining build_locally=0 and scheduler backends that cannot reliably retrieve the job's exit code (#3426)

Bug fixes

• Correctly handle verbatim scheduler options passed in the CLI in the remote auto-detection script (#3456)
• Treat correctly failed_stage=None in failure stats report (#3454)
• Fix constraint evaluation for Slurm features starting with numbers (#3448)
• Fix lock order when storing reports to the database (#3450)
• Update special parameter character in test name (#3441)
• Disallow command options from execution modes (#3432)
• Fix calculation of lower/upper reference bounds when their absolute value reaches zero (#3433)
• Insert new line between stdout/stderr listings in failures (#3429)
• Disable number parsing when in --list-stored-sessions (#3425)

Other

• Add GH action to test EOL python versions (#3417)
• Enabled slurmdbd in Docker compose setup (#3391)
• Use Python 3.13 to publish to PyPI (#3406)
• Update YAML unit test config (#3405)
• Replace symlinks with file copies in Docker compose config for Github workflows (#3396)
• Remove stale Dockerfiles (#3387)
• Update Slurm cluster images to ubuntu:22.04 (#3383)