feat(tests): manually run functional tests on pr#20580
Conversation
There was a problem hiding this comment.
Pull request overview
This PR reduces CircleCI usage by gating PR functional (Playwright) tests behind a pipeline parameter and providing manual triggers (local script + Claude skill) to run them on demand.
Changes:
- Add
enable_functional_testspipeline parameter and move PR functional tests into a separate conditional workflow. - Add
_scripts/trigger-functional-tests.shto trigger the manual workflow and cancel prior runs on the same branch. - Add a Claude skill (
/run-functional-tests) that guides users through invoking the trigger script.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
.claude/skills/run-functional-tests/SKILL.md |
Adds a user-invocable Claude skill documenting how to trigger manual PR functional tests. |
.circleci/config.yml |
Introduces a new pipeline parameter and conditional workflow for manual functional test runs; removes functional tests from default PR workflow. |
_scripts/trigger-functional-tests.sh |
Adds a branch-aware CircleCI trigger script that starts a functional-only pipeline and cancels prior functional workflows on the same branch. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
4b338dc to
cdb68d9
Compare
af7947a to
23528d7
Compare
vbudhram
left a comment
There was a problem hiding this comment.
@nshirley what are your thoughts on running the functional tests via branch name (ie anything containing ci), or even commit message containing ([run-ci]). That way we don't need a skill or bash script.
Kinda leaning towards commit message since that gives good flexibility.
|
@vbudhram I suppose that could work too. I'd be worried we would forget to add the CI in the branch name, and the commit message means we'd have to commit and push a unique message to run tests, then rebase after tests pass and push again, then wait for build/unit/integration again. We still want to 'require' functional tests before merging, just don't want to run them on every single push This approach just adds an Here's one that I approved, but that page just shows a button to approve it if it's pending. |
23528d7 to
cd2ac5c
Compare
| # a friendly message on HTTP errors so callers don't have to interpret raw | ||
| # error responses (e.g. an HTML 401 body that would otherwise blow up jq). | ||
| # Usage: circleci_api METHOD URL [extra-curl-args...] | ||
| circleci_api() { |
There was a problem hiding this comment.
This is gross. Let's have the user install circleci cli tool. They should have it anyways as there other benefits especially when working with Claude.
|
|
||
| set -euo pipefail | ||
|
|
||
| for cmd in curl jq git; do |
There was a problem hiding this comment.
We could require the circleci command line utility instead, which I think would be better. Since these command line tools are so helpful, I think we should make them required as part of the development setup. I'll update our internal docs accordingly.
dschom
left a comment
There was a problem hiding this comment.
Would it be a big deal to update the script to use the circleci cli tool. I think this will be result in a shorter more robust script.
|
@dschom no issue updating this! I have the cli installed too but API felt more flexible, but I'll re-work it to clean it up! |
Because: - We want to reduce the ci credit usage from running functional tests on every pr push This pull request: - Gates PR functional tests behind a ci parameter - Adds a script to trigger tests locally for a given branch - Adds a Claude skill to help run the tests as well Closes: FXA-13697
cd2ac5c to
2eef1af
Compare
|
@dschom, following up here from our thread. The CLI does not provide the ability to approve pending workflows and it appears the majority of functionality is supposed to go through the API. To that, I did my best to clean the script up and make it a bit nicer - it was definitely over engineered |

Because
This pull request
Issue that this pull request solves
Closes: FXA-13697
Checklist
Put an
xin the boxes that applyHow to review (Optional)
This makes it so the Functional Tests for PR are only run with an approval. As such, there's now a script which you can run manually, or a Claude skill which calls the script. The nice thing about the skill is that we can use it with loop skill to have Claude check if approval is ready and then wait and try again!
Screenshots (Optional)
Please attach the screenshots of the changes made in case of change in user interface.
Other information (Optional)
Any other information that is important to this pull request.