Feature Request
How can we make CDash better?
Our project has a massive combinatorial matrix of configuration-time options (different parallel backends, compilers, hardware architectures, and feature flags). For every Pull Request, we run 50+ builds, and our nightly regression suites are scattered across a number of different distributed sites.
Currently, we are struggling to keep track of our overall configuration coverage. While CDash is excellent at showing the health of individual builds in a list, it is very difficult to quickly answer macro-level questions like:
- Are we actually testing the specific combination of
Compiler X + Backend Y + Feature Flag Z in this PR or nightly cycle?
- Did a recent infrastructure change inadvertently drop coverage for a rare architecture combination across our sites?
Relying on a flat list of 50+ builds or parsing long, complex string names in the "Build Name" field makes it incredibly easy to miss gaps in our testing matrix.
A solution we'd like
We are looking for a way to track and visualize this multi-dimensional build matrix directly within CDash, so we can treat "configuration coverage" as a first-class metric alongside traditional line coverage.
Before proposing a specific UI design or implementation, we wanted to ask if CDash currently supports a way to achieve this, or if there is an intended best practice for this use case? For example, is there an existing mechanism (perhaps leveraging custom build properties, build attributes, or specific dashboard filters) that would allow us to aggregate and view our builds as a dynamic matrix or pivot table rather than a flat list?
Alternatives we've considered
- Strict build naming conventions: We currently pack some of the configuration attributes into the build name string, but this doesn't allow for automated gap analysis or clean visualization. Packing all of it would be prohibitively expensive.
- Custom external dashboards: Scraping the CDash API to reconstruct and visualize the matrix in a separate internal tool, though we would highly prefer to keep CDash as our single source of truth for project health. We have been looking at uploading our
CMakeCache.txt files via CTEST_NOTES_FILES but that looks more like a stop gap than a real solution.
Feature Request
How can we make CDash better?
Our project has a massive combinatorial matrix of configuration-time options (different parallel backends, compilers, hardware architectures, and feature flags). For every Pull Request, we run 50+ builds, and our nightly regression suites are scattered across a number of different distributed sites.
Currently, we are struggling to keep track of our overall configuration coverage. While CDash is excellent at showing the health of individual builds in a list, it is very difficult to quickly answer macro-level questions like:
Compiler X+Backend Y+Feature Flag Zin this PR or nightly cycle?Relying on a flat list of 50+ builds or parsing long, complex string names in the "Build Name" field makes it incredibly easy to miss gaps in our testing matrix.
A solution we'd like
We are looking for a way to track and visualize this multi-dimensional build matrix directly within CDash, so we can treat "configuration coverage" as a first-class metric alongside traditional line coverage.
Before proposing a specific UI design or implementation, we wanted to ask if CDash currently supports a way to achieve this, or if there is an intended best practice for this use case? For example, is there an existing mechanism (perhaps leveraging custom build properties, build attributes, or specific dashboard filters) that would allow us to aggregate and view our builds as a dynamic matrix or pivot table rather than a flat list?
Alternatives we've considered
CMakeCache.txtfiles viaCTEST_NOTES_FILESbut that looks more like a stop gap than a real solution.