docs(weave): add Compare evaluations page (DOCS-2540)#2931
Conversation
Add a new How-to guide under Evaluate your agents and applications that documents the Weave Compare evaluations view: how to open it, what the Summary and Dataset results tabs show, and how to set a baseline, reorder evaluations, and remove an evaluation from the comparison. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resume prompt (for a fresh Claude Code session)
|
|
Images automagically compressed by Calibre's image-actions ✨ Compression reduced images by 75%, saving 427.9 KB.
|
|
Images automagically compressed by Calibre's image-actions ✨ Compression reduced images by 10.8%, saving 6.8 KB.
3 images did not require optimisation. |
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
📚 Mintlify Preview Links✨ Added (1 total)📄 Pages (1)
📝 Changed (1 total)⚙️ Other (1)
🤖 Generated automatically when Mintlify deployment succeeds |
🔗 Link Checker ResultsPreview: https://wb-21fd5541-compare-eval-docs-2540.mintlify.site |
HiveMind Sessions1 session · 23m · $5.54
Screenshots
View all sessions in HiveMind → Run |
… badge color (DOCS-2540) Verified against wandb/core CompareEvaluationsPage source: - Dataset results tab is a per-example table (dataset input + each evaluation's output and scorer metrics) with a row-detail pane; when comparing two evaluations it shows two scatter plots for brushing to filter diverging examples. Document the empty (no shared rows) state. - Match Dataset/Scoring inconsistency warnings to the source: the dataset/scorer varies across evaluations; examples are limited to the dataset intersection. - Badge color follows metricLowerIsBetter (higher-is-better by default), not a fixed improve/regress rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
|
||
| The W&B Weave **Compare evaluations** view lets you inspect two or more logged evaluations side by side. Use it to see how metrics move between runs when you change a prompt, model, or scorer, and to catch dataset or scorer inconsistencies before you draw conclusions from the results. | ||
|
|
||
| This guide is for Weave users who want to compare logged evaluations in the Weave UI. It covers how to open the Compare evaluations view, what appears in the summary charts and results tables, and the actions you can take to tailor the comparison, including setting a baseline, reordering evaluations, and removing an evaluation from the comparison. |
There was a problem hiding this comment.
Is this section a part of the authoring skill now? I've noticed the bot using it a lot. I feel like it fluffily rehashes the introduction and I'm not a big fan of it.
There was a problem hiding this comment.
no, the author skill hasn't changed, perhaps its model drift causing different behavior.
tho, fwiw, although i agree it is 'fluffy' - it didn't strike me as something abnormal for docs - typically pages have a introductory recap of "what info will you find in this page?". no? Do we not want that anymore?
| * **Properties**: Any additional properties recorded on the evaluation. | ||
| * **Metrics**: The aggregate value for each metric produced by the evaluation's scorers. | ||
|
|
||
| For numeric metrics, the difference from the baseline is displayed as a colored badge next to the value. The color reflects whether the change is in the favorable or unfavorable direction for that metric: by default, higher values are treated as better, so a green badge marks an increase and a red badge marks a decrease. For metrics configured as lower-is-better, the colors are reversed. |
2 participants
Summary
Adds a new How-to guide, Compare evaluations, to the Weave docs. It documents the Compare evaluations view: opening it from the Evals table, reading the Summary and Dataset results tabs, and the customization actions (set a baseline, change comparison order, remove from comparison).
weave/guides/evaluation/compare_evals.mdxweave/guides/evaluation/img/docs.json(English nav only)Resolves DOCS-2540.
What the ticket asked for (and how it's handled)
img/— done.compare_evals.mdx, title "Compare evaluations" — done.Verified against source (
wandb/core)Several claims were checked directly against
frontends/weave/.../CompareEvaluationsPage:ScorecardSection.tsx): "Dataset inconsistency detected" fires when the dataset ref varies across evals and limits examples to the dataset intersection; "Scoring inconsistency detected" fires when the scorer ref varies. Page wording updated to match.CompareEvaluationsPage.tsx,ExampleCompareSection,ExampleFilterSection): per-example table with a row-detail pane; two scatter plots (brush-to-filter) render only when comparing exactly two evaluations; shows a "0 rows in common" alert when datasets don't intersect.ScorecardSection.tsx): driven by ametricLowerIsBetterflag — higher-is-better by default (increase = green), reversed for lower-is-better metrics. Page wording updated from the earlier improve/regress phrasing.Recommendations for technical review
Technical accuracy
metricLowerIsBetter. In the ticket screenshot,violation_countshows green on an increase because it isn't flagged lower-is-better. Confirm whether that's expected (built-in scorers may not set the flag), so we can decide whether the page needs a caveat.Nice to have
Sources
weave/guides/tools/comparison.mdx.wandb/corefrontends/weave/src/components/PagePanelComponents/Home/Browse3/pages/CompareEvaluationsPage/.🤖 Generated with Claude Code