Hi folks,
Thank you for the new custom reporter planning API!
We tried it out as part of the our shard balancing algorithm in the https://github.com/flakiness/playwright. Our balancer relies on the previous test durations to allocate well-balanced shards.
With the new API surface, we can't properly measure the weight of the setup / teardown projects since we're not given the test cases in these projects:
fullConfig exposes the project dependency tree, but not the test cases inside these projects that will be executed- the
suite exposes only top-level project suites and their test cases, not the test cases inside setup/teardown projects
As a result, a commit that drops / adds some tests in the setup or teardown projects will not result in shard re-balancing.
Any chance the API can be enhanced to expose more information to clients?
Thanks,
Andrey
P.S. As of today, we're able to get tests from the setup/teardown projects: we have the flakiness-playwright-shard binary that runs playwright with --list to get all the test cases scheduled to run, then fetches durations for them, and then allocates shards. It'd be nice to match this behavior to get rid of the multiple hops we currently do.
Hi folks,
Thank you for the new custom reporter planning API!
We tried it out as part of the our shard balancing algorithm in the https://github.com/flakiness/playwright. Our balancer relies on the previous test durations to allocate well-balanced shards.
With the new API surface, we can't properly measure the weight of the setup / teardown projects since we're not given the test cases in these projects:
fullConfigexposes the project dependency tree, but not the test cases inside these projects that will be executedsuiteexposes only top-level project suites and their test cases, not the test cases inside setup/teardown projectsAs a result, a commit that drops / adds some tests in the setup or teardown projects will not result in shard re-balancing.
Any chance the API can be enhanced to expose more information to clients?
Thanks,
Andrey
P.S. As of today, we're able to get tests from the setup/teardown projects: we have the
flakiness-playwright-shardbinary that runs playwright with--listto get all the test cases scheduled to run, then fetches durations for them, and then allocates shards. It'd be nice to match this behavior to get rid of the multiple hops we currently do.