Skip to content

40-reporting.md

Latest commit

 

History

History
467 lines (325 loc) · 11.4 KB

40-reporting.md

File metadata and controls

467 lines (325 loc) · 11.4 KB

Runner and Reporting v0 Draft

1. Role

The runner evaluates normalized suite core, plus normalized inventory when inventory-driven cases are present, and emits a machine-readable report.

This document defines the suite-runner report.

Product-certification reporting is specified separately in 60-product-certification.md.

The report is part of the platform contract, not an incidental output format.

2. Required Runner Outcomes

The runner MUST determine:

  • selected cases
  • case execution order
  • action events within each case
  • artifact events within each case
  • assertion outcomes within each case
  • overall process exit status

3. Case Ordering

Cases MUST be ordered deterministically by:

  1. suite item order in source-normalized order
  2. case-source-defined canonical case order

This ordering rule defines report order as well.

For inventory-driven items, canonical case order is canonical inventory order.

For workflow-driven items, canonical case order is the explicit deterministic order defined by the discovery source.

4. Case Identity

Every case record MUST contain a stable case identity.

At minimum that identity MUST include:

  • suite item id
  • case key

The report MUST NOT rely on test name alone as case identity.

If a case originates from inventory selection, the report MUST also emit the selected inventory test name explicitly.

5. Canonical and Volatile Fields

Reports MAY contain both canonical and volatile fields.

Canonical fields are expected to be stable across equivalent runs. Volatile fields may vary even when the logical outcome is unchanged.

Canonical fields include:

  • case identity
  • case status
  • assertion counts
  • action kinds
  • fail classifications

Volatile fields include, unless a stable mode overrides them:

  • wall-clock timestamps
  • elapsed durations
  • host-specific execution metadata

Non-canonical analytics metadata is described separately in 41-analytics.md. Unless explicitly enabled, analytics metadata is outside the required canonical record set.

Derived analytical build packaging is described separately in 43-build-analytics.md. That package is downstream of the report and does not alter report semantics.

If volatile fields are emitted, they SHOULD be explicitly labeled or documented as non-canonical.

6. Report Format

A structured line-oriented machine format is required for v0. JSONL is the canonical format.

JSON records MUST be UTF-8 encoded.

In golden mode, object keys MUST be emitted in ascending lexicographic order by raw UTF-8 byte sequence of the key strings.

A report MUST include:

  • one header record
  • one case summary record per case
  • zero or more assertion records
  • zero or more action records
  • zero or more artifact records
  • zero or more extraction records
  • one final summary record

Optional non-canonical telemetry records may be emitted only through an explicit analytics lane as described in 41-analytics.md.

Human-facing console presentation is specified separately in 42-console-ui.md. That console is a presentation layer over the same report events, not a separate execution or reporting semantics.

Derived analytical packaging is specified separately in 43-build-analytics.md. That analytical package is a mechanical projection over report and telemetry records, not a second execution or reporting semantics.

6.1 Header Record

The first record MUST be the header record:

{
	"k": "observer_report",
	"v": "0"
}

Required canonical fields:

  • k: fixed string observer_report
  • v: spec version string
  • inventory_sha256: SHA-256 of normalized inventory bytes
  • suite_sha256: SHA-256 of normalized suite bytes
  • mode: default or golden

The normalized byte sequences for those hash fields are defined in 25-normalization.md.

If a suite does not use inventory-driven case sources, inventory_sha256 MUST still be emitted as the SHA-256 of the canonical empty inventory byte sequence so report shape remains uniform.

Optional volatile fields:

  • generated_at_utc
  • host
  • implementation

6.2 Case Identifier

The canonical case identifier is a stable string derived from the tuple:

  • suite item id
  • case key

The derivation algorithm is:

  1. encode item_id as UTF-8 bytes
  2. append byte 0x1f
  3. append UTF-8 bytes of case_key
  4. encode the resulting byte sequence using unpadded Base64URL

This string is emitted as case_id.

Reports MUST also emit item_id and case_key explicitly; consumers MUST NOT be required to reverse case_id to recover canonical identity.

For inventory-driven cases, reports MUST additionally emit test_name, where test_name = case_key.

6.3 Case Summary Record

Each finished case MUST emit one case summary record:

{
	"k": "case",
	"case_id": "...",
	"item_id": "item-1",
	"case_key": "Smoke::Version",
	"test_name": "Smoke::Version",
	"status": "pass"
}

Required canonical fields:

  • k: fixed string case
  • case_id
  • item_id
  • case_key
  • status: pass or fail
  • assert_pass
  • assert_fail
  • unhandled_action_fail

Required for inventory-driven cases only:

  • test_name

Optional canonical fields for workflow-oriented cases:

  • case_inputs: bounded canonical summary of discovered input fields

Optional volatile fields:

  • duration_ms
  • started_at_utc
  • ended_at_utc

Optional bounded diagnostic fields:

  • notes

6.4 Assertion Record

Each evaluated assertion SHOULD emit one assertion record:

{
	"k": "assert",
	"case_id": "...",
	"assert_ix": 0,
	"status": "pass",
	"msg": "expectation passed"
}

Required canonical fields:

  • k: fixed string assert
  • case_id
  • assert_ix: zero-based assertion index within the case
  • status: pass or fail
  • msg

Optional fields:

  • loc: object with line and col
  • predicate

6.5 Action Record

Each attempted action SHOULD emit one action record:

{
	"k": "action",
	"case_id": "...",
	"action_ix": 0,
	"action_id": "compile",
	"action": "run",
	"status": "ok"
}

Required canonical fields:

  • k: fixed string action
  • case_id
  • action_ix: zero-based action index within the case
  • action: stable action kind string such as run, proc, httpGet, tcp, artifactPublish, artifactCheck, extractJson, or extractJsonl
  • status: ok or fail
  • args: bounded canonical action argument summary

Optional canonical fields:

  • action_id: stable action identifier within the suite item when one is declared
  • artifact_in: array of objects with name and kind
  • artifact_out: array of objects with name and kind

Exactly one of these payloads MUST be present:

  • ok
  • fail

ok payload shapes:

  • for run and proc:
    • exit
    • out_len
    • err_len
    • optional out_preview_b64
    • optional err_preview_b64
    • optional out_truncated
    • optional err_truncated
  • for httpGet:
    • status
    • body_len
    • optional body_preview_b64
    • optional body_truncated
  • for tcp:
    • bytes_len
    • optional bytes_preview_b64
    • optional bytes_truncated
  • for artifactPublish and artifactCheck:
    • name
    • kind
    • path
  • for extractJson and extractJsonl:
    • source_artifact
    • select
    • match_count
    • optional value_text
    • optional value_text_truncated

fail payload shape:

  • kind
  • msg
  • optional code

Optional volatile fields:

  • duration_ms

6.6 Artifact Record

Each explicit artifact publication or artifact check SHOULD emit one artifact record:

{
	"k": "artifact",
	"case_id": "...",
	"artifact_ix": 0,
	"action_ix": 0,
	"name": "typed_unit",
	"event": "publish",
	"kind": "jsonl",
	"status": "ok"
}

Required canonical fields:

  • k: fixed string artifact
  • case_id
  • artifact_ix: zero-based artifact record index within the case
  • action_ix: producing or checking action index within the case
  • name: canonical artifact binding name
  • event: publish or check
  • kind: stable artifact kind string
  • status: ok or fail

Optional canonical fields:

  • location: object with canonical path
  • producer_action_id
  • consumer_action_id

For publish events, location SHOULD be present.

For check events, the record MAY additionally include:

  • observed_exists: boolean encoded as true or false

If status is fail, the record SHOULD include a canonical fail payload with stable kind and msg fields.

6.7 Extraction Record

Each explicit structured extraction SHOULD emit one extraction record:

{
	"k": "extract",
	"case_id": "...",
	"extract_ix": 0,
	"action_ix": 1,
	"source_artifact": "typed_unit",
	"format": "jsonl",
	"status": "ok"
}

Required canonical fields:

  • k: fixed string extract
  • case_id
  • extract_ix: zero-based extraction index within the case
  • action_ix: extraction action index within the case
  • source_artifact: artifact binding name consumed by the extraction
  • format: stable extraction format string
  • status: ok or fail

Optional canonical fields:

  • select: declared selector string
  • summary: bounded canonical extraction summary with:
    • match_count
    • optional value_text
    • optional value_text_truncated

If status is fail, the record SHOULD include a canonical fail payload with stable kind and msg fields.

6.8 Final Summary Record

The last record MUST be a summary record:

{
	"k": "summary",
	"case_pass": 1,
	"case_fail": 0,
	"exit_code": 0
}

Required canonical fields:

  • k: fixed string summary
  • case_pass
  • case_fail
  • assert_pass
  • assert_fail
  • exit_code

6.9 Report As Source Fact Stream

The report is the authoritative structured fact stream for one completed build.

Downstream consumers MAY derive secondary artifacts from that stream, including:

  • local console presentation
  • telemetry summaries
  • analytical build packages
  • workflow artifact summaries

Those derived artifacts MUST be mechanically derived from explicit report and telemetry records.

They MUST NOT redefine:

  • case identity
  • action identity
  • pass or fail semantics
  • runner exit semantics

If a downstream analytical package is produced, the report remains the canonical source artifact and the analytical package remains derivative.

7. Minimum Case Summary Shape

A case summary MUST contain:

  • case id
  • item id
  • case key
  • status
  • assert pass count
  • assert fail count
  • unhandled action fail count

If the case is inventory-driven, it MUST also contain:

  • test name

It MAY also contain:

  • case input summary
  • notes
  • durations
  • truncated previews

8. Payload Discipline

Reports MUST avoid dumping unbounded payloads by default.

Large output bodies SHOULD be represented by:

  • total size
  • explicit truncation markers
  • bounded preview fields when needed

Preview fields that carry bytes MUST use unpadded Base64URL.

9. Runner Exit Status

The runner MUST use this exit status model:

  • 0: all cases passed
  • 1: one or more cases failed
  • 2: runner or specification error

10. Golden Mode Guidance

If Observer is used in golden workflows, the implementation SHOULD provide a mode that excludes or normalizes volatile fields.

Golden mode does not change semantics. It changes serialization discipline so that logically equivalent runs are diff-stable.

Golden mode SHOULD omit volatile fields when feasible. If a volatile field cannot be omitted, it SHOULD be normalized to a stable sentinel value.