Skip to content

Performance: add dry-run mode and sanity-range assertions to CodeVitals posting #49999

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
LiamSarsfield wants to merge 10 commits into
base: trunk
Choose a base branch
from
+887 −69
Draft

Performance: add dry-run mode and sanity-range assertions to CodeVitals posting #49999

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
24ef48c
Performance: add dry-run mode and sanity-range assertions to CodeVita…
LiamSarsfield Jun 26, 2026
06f0dd3
Performance: default CodeVitals POST to the apex host
LiamSarsfield Jun 26, 2026
4587f3e
Performance: fail the CodeVitals sanity guard closed + add unit coverage
LiamSarsfield Jun 26, 2026
d4a0a21
Performance: fix direct-run guard, gate unit tests, broaden coverage
LiamSarsfield Jun 26, 2026
1b6a50b
Performance: keep the CodeVitals token out of logs and errors
LiamSarsfield Jun 26, 2026
bf53477
Performance: scrub the CodeVitals token from error cause chains
LiamSarsfield Jun 26, 2026
e02152c
Merge remote-tracking branch 'origin/trunk' into add/codevitals-dry-r…
LiamSarsfield Jun 26, 2026
d6c8e8b
Performance: fail closed when a keyed CodeVitals metric omits its type
LiamSarsfield Jun 26, 2026
37c235c
Performance: make local validation failures always fail the CodeVital…
LiamSarsfield Jun 26, 2026
648b630
Performance: make scenario misconfig fail closed and scrub string err…
LiamSarsfield Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 55 additions & 18 deletions tools/performance/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,16 @@ The test suite is designed to run in TeamCity. See `TEAMCITY-SETUP.md` for detai

### Environment Variables

| Variable | Description |
|----------|-------------|
| `CODEVITALS_TOKEN` | API token for posting results to CodeVitals |
| `CODEVITALS_URL` | CodeVitals API URL (default: https://www.codevitals.run) |
| `COMPOSE_PROJECT_NAME` | Unique Docker project name for build isolation |
| `GIT_COMMIT` | Git commit SHA for tracking (auto-detected from plugin) |
| `GIT_BRANCH` | Git branch for tracking (default: trunk) |
| `ITERATIONS` | Number of measurement iterations (default: 5) |
| `WP_ADMIN_USER` | WordPress admin username (default: admin) |
| `WP_ADMIN_PASS` | WordPress admin password (default: password) |
| Variable | Description |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `CODEVITALS_TOKEN` | API token for posting results to CodeVitals |
| `CODEVITALS_URL` | CodeVitals API URL (default: https://codevitals.run). Use the apex host, not `www.`: the `www.` host 301-redirects the API and the redirect drops the POST body. Set an origin-only URL (scheme + host); the API path is appended, so any path prefix on this value is not preserved. |
| `COMPOSE_PROJECT_NAME` | Unique Docker project name for build isolation |
| `GIT_COMMIT` | Git commit SHA for tracking (auto-detected from plugin) |
| `GIT_BRANCH` | Git branch for tracking (default: trunk) |
| `ITERATIONS` | Number of measurement iterations (default: 5) |
| `WP_ADMIN_USER` | WordPress admin username (default: admin) |
| `WP_ADMIN_PASS` | WordPress admin password (default: password) |

## Metric

Expand All @@ -41,11 +41,48 @@ The test suite is designed to run in TeamCity. See `TEAMCITY-SETUP.md` for detai

## Scripts

| Script | Description |
|--------|-------------|
| `pnpm test` | Run full test suite (auto-clones plugin if needed) |
| `pnpm test:quick` | Quick test with 2 iterations |
| `pnpm calibrate` | Run CPU throttling calibration |
| `pnpm measure` | Run LCP measurement only |
| `pnpm report` | Post results to CodeVitals only |
| `pnpm test -- --skip-codevitals` | Run tests without posting to CodeVitals |
| Script | Description |
| -------------------------------- | ---------------------------------------------------------------------- |
| `pnpm test` | Run full test suite (auto-clones plugin if needed) |
| `pnpm test:quick` | Quick test with 2 iterations |
| `pnpm calibrate` | Run CPU throttling calibration |
| `pnpm measure` | Run LCP measurement only |
| `pnpm report` | Post results to CodeVitals only |
| `pnpm report:dry` | Build and print the CodeVitals payload without posting (CI smoke test) |
| `pnpm test -- --skip-codevitals` | Run tests without posting to CodeVitals |

## Safeguards

CodeVitals is an **append-only** store with no self-service rollback. Once a bad point lands (wrong key, out-of-range value, scale error), the trend graph stays polluted until a CodeVitals admin corrects it. The safeguards below keep bad data out.

### Dry run

`pnpm report:dry` builds the full payload, prints it, and exits without posting. It needs no `CODEVITALS_TOKEN`, so it works as a CI smoke test. Use it to inspect a payload before a real `pnpm report`.

### Sanity-range assertions

`post-to-codevitals.js` checks every typed metric against `SANITY_RANGES` in `scenarios.js` before posting. A value outside its range is logged and skipped (not posted), and the script exits non-zero so CI surfaces the failure. Other valid metrics in the same run still post.

| Metric | Min | Max |
| ------ | --- | ----- |
| `lcp` | 100 | 60000 |
| `ttfb` | 10 | 10000 |
| `fcp` | 50 | 30000 |
| `tbt` | 0 | 10000 |
| `cls` | 0 | 5 |

Add a row when a new metric type starts being posted, and set `metricType` on the scenario so the check applies to it.

### Staging keys

Post a new metric to a `-staging` CodeVitals key first (e.g. `…-timeToFirstByte-staging`) for 2-3 builds. Inspect the values in the CodeVitals UI, then rename to the production key. This gives a safety window before a new metric reaches production.

### If bad data lands anyway

A bad point must be corrected by the CodeVitals admin. Steps:

1. **Stop posting.** Pause the CodeVitals Jetpack Performance Scheduler build in TeamCity.
2. **Document the extent.** Record the affected metric keys, the commit range (monorepo hashes), the time window (build start to end), and whether the values are isolated or systematic.
3. **Request a correction.** CodeVitals runs outside this project; send the request through the team channel named in the FORMS-696 runbook. Include metric ID 113, the affected keys, the commit/timestamp range, and the root cause.
4. **Fix the root cause.** Add or tighten a sanity range or staging gate. Don't re-enable the Scheduler until the fix merges.
5. **Record the incident.** Add the failure mode, detection timing, and prevention measures to the FORMS-696 maintenance runbook.
6 changes: 4 additions & 2 deletions tools/performance/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,11 @@
"docker:up": "docker compose -f docker/docker-compose.yml up -d",
"measure": "node scripts/measure-lcp.js",
"report": "node scripts/post-to-codevitals.js",
"report:dry": "node scripts/post-to-codevitals.js --dry-run",
"setup:browsers": "playwright install chromium",
"test": "node scripts/run-performance-tests.js",
"test:quick": "ITERATIONS=2 node scripts/run-performance-tests.js"
"test": "node --test && node scripts/run-performance-tests.js",
"test:quick": "ITERATIONS=2 node scripts/run-performance-tests.js",
"test:unit": "node --test"
},
"dependencies": {
"dotenv": "^16.3.1",
Expand Down
Loading
Loading