Skip to content

[DOCS]: Add portable regression receipt example to results-and-reporting#70

Open
sumit1kr wants to merge 2 commits into
microsoft:mainfrom
sumit1kr:docs/regression-receipt-example
Open

[DOCS]: Add portable regression receipt example to results-and-reporting#70
sumit1kr wants to merge 2 commits into
microsoft:mainfrom
sumit1kr:docs/regression-receipt-example

Conversation

@sumit1kr

@sumit1kr sumit1kr commented Jun 2, 2026

Copy link
Copy Markdown

Closes #63

Adds a "Portable Regression Receipt" section to docs/usage/results-and-reporting.md
showing how to use existing report.metadata and result.metadata fields to produce
a stable, diffable CI artifact tied to JsonFileReportSink.

No new Python types introduced — docs only.

@sumit1kr sumit1kr requested review from a team and Copilot June 2, 2026 11:41

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds documentation describing a "Portable Regression Receipt" pattern for CI gating using JsonFileReportSink, including a sample JSON artifact illustrating run-level and scenario-level metadata.

Changes:

  • Adds a new section to docs/usage/results-and-reporting.md explaining how to use report.metadata and result.metadata for stable regression artifacts.
  • Provides an example JSON receipt with sample fields.

@sumit1kr

sumit1kr commented Jun 2, 2026

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

@gitrishiom

Copy link
Copy Markdown

@sumit1kr

Apologies for the slow response, and thanks for taking this on.

I agree with the docs-only direction here. I also agree with keeping the broader runtime-evidence / production-receipt idea from #63 out of scope for this PR; this section works best as test-time CI evidence using RAMPART’s existing reporting surfaces.

One small thing I noticed while re-checking the current sink behavior: TestRunReport has metadata, but JsonFileReportSink._serialize_report() does not currently include report.metadata in the emitted JSON. It serializes the aggregate fields, population_summary, by_harm_category, and each result’s metadata.

Could you update the docs example to mirror the current JsonFileReportSink output shape and put the stable receipt fields under result.metadata / the relevant by_harm_category result entry?

If maintainers would rather support run-level metadata in the built-in JSON output, then the alternate path would be a tiny serialization + test change for report.metadata, but my preference is to keep this PR docs-only and avoid scope creep.

Once that small alignment is made, I’m happy to take another pass on the diff. Thanks again for moving this forward.

@sumit1kr sumit1kr force-pushed the docs/regression-receipt-example branch from 4ae7e65 to d9f3146 Compare June 29, 2026 09:56
@sumit1kr

Copy link
Copy Markdown
Author

@gitrishiom Updated — moved the receipt fields under result.metadata (nested inside the relevant by_harm_category entry) to match the current JsonFileReportSink output shape, and added a note clarifying that report.metadata isn't currently serialized. Pushed as a follow-up commit. Ready for another pass whenever you get a chance — thanks for the detailed review!

@spencrr tagging again since this is ready for maintainer review/approval whenever you have bandwidth.

@gitrishiom

Copy link
Copy Markdown

Thanks, this addresses the concern.

The example now matches the current JsonFileReportSink output shape, and the note about report.metadata not being serialized is helpful. From my side, this looks aligned with the docs-only scope for #63.

I’ll leave final review/approval to @spencrr . Thanks again for the quick update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docs: example pattern for agent safety regression receipts in reporting sinks

3 participants