50-workflow-verification.md

Workflow Verification v0 Proposal

1. Role

This document defines a proposed Observer model for multi-stage local workflow verification.

Its target problem is not ordinary unit testing.

Its target problem is the recurring class of local scripts that:

• discover input fixtures or corpus cases
• run a sequence of build or analysis tools
• materialize intermediate artifacts
• feed those artifacts into later stages
• assert over structured results, files, exit codes, or output text
• persist enough evidence that failures can be explained later

Observer v0 already models inventory-driven test execution well.

This proposal addresses the harder and less solved problem: deterministic verification of staged artifact workflows on a local machine.

2. Non-Goal

This proposal is not a generic shell task runner.

It is not CI YAML in another syntax.

It is not a replacement for remote orchestration systems.

The goal is narrower and stronger:

• local workflow verification
• artifact-driven stage boundaries
• deterministic execution and evidence
• canonical reporting over staged derivation

3. Problem Statement

Many real verification flows are currently encoded as one-off scripts.

Those scripts usually mix four concerns in one place:

• case discovery
• tool wiring
• staged execution
• verdict logic

That creates recurring problems:

• no canonical model for what a case is
• no reusable way to bind artifacts between stages
• no stable distinction between workflow failure and subject failure
• no standard report or evidence format
• no compatibility between scripts that solve nearly the same problem
• proliferation of local automation that is hard to trust, diff, or retire

Observer should absorb that burden with one explicit workflow model.

4. Core Thesis

Workflow verification should be modeled as first-class structured execution.

A workflow case is not just one run action.

It is an ordered sequence or graph of explicit actions where:

• each action has declared inputs
• each action has declared outputs
• later actions consume named artifacts from earlier actions
• assertions operate on structured action results or explicit artifacts
• evidence for the whole workflow is captured through one canonical report

5. Core Model

5.1 Workflow Definition

A workflow definition contains:

• workflow id
• case source
• case order rule
• action plan
• assertion plan
• output retention policy

5.2 Workflow Case

A workflow case contains:

• case key: stable identity within the workflow input domain
• case inputs: explicit source values used to instantiate actions
• action instances: deterministic ordered action executions for that case
• assertions: explicit checks over action results and artifacts

The canonical case identity remains a stable pair of:

• workflow item id
• case key

5.3 Action Instance

Each action instance contains:

• action id: stable name within the workflow item
• kind
• explicit arguments
• explicit declared inputs
• explicit declared outputs
• result status
• retained evidence summary

Action kinds may include existing Observer action kinds such as:

• run
• proc
• httpGet
• tcp

Workflow verification also requires artifact-aware actions such as:

• file existence checks
• file parse steps
• JSON extraction
• directory discovery
• explicit artifact publication

6. Case Discovery

Workflow verification MUST support explicit case discovery.

Discovery is part of the contract, not an implementation detail.

Initial case-source forms should be explicit and local-machine deterministic, such as:

• enumerate files under a root with a declared pattern
• enumerate rows from a declared manifest file
• enumerate cases from a previously published JSON artifact

Discovery MUST define:

• the stable case key
• the explicit case input fields
• the deterministic case order

If discovery depends on the filesystem, the resulting case set MUST be ordered deterministically by explicit path normalization rules.

7. Artifact Model

Artifacts are first-class workflow values.

An artifact is not just an incidental file path string.

An artifact binding contains at minimum:

• artifact name
• producer action id
• kind
• location
• existence status

Representative artifact kinds:

• file
• directory
• json
• jsonl
• bytes
• text

Later actions MUST consume artifacts through named bindings, not by heuristic path reconstruction.

Observer MUST NOT infer downstream artifact paths from free-form command strings.

8. Determinism Rules

Workflow verification is governed by the same top-level priorities as the rest of Observer.

Additional workflow-specific rules are:

1. case discovery order MUST be deterministic
2. action order within a case MUST be deterministic
3. artifact bindings MUST be explicit
4. downstream stages MUST consume named prior outputs, not guessed paths
5. verdict logic MUST be explicit and mechanically derivable from action results and artifact facts

If an action depends on workspace state outside declared inputs, that dependency MUST be explicit in the workflow definition.

9. Failure Model

Workflow verification requires a clearer failure split than most ad hoc scripts provide.

At minimum, Observer should distinguish:

• runner failure: Observer could not execute the workflow contract
• infrastructure failure: a declared action could not be performed as specified
• workflow failure: a declared stage completed but violated an explicit workflow expectation
• assertion failure: an explicit final or intermediate assertion failed
• invariant failure: a required workflow validity condition did not hold

This distinction is required for meaningful reports.

For example:

• missing required executable path is infrastructure failure
• command returned nonzero when zero was required is workflow failure
• expected output file was absent is workflow failure
• parsed JSON did not contain required field value is assertion failure
• upstream artifact published under wrong kind is invariant failure

10. Reporting Extension

The existing report model in 40-reporting.md already supports cases, actions, and assertions.

Workflow verification should extend that model rather than replace it.

Additional report facts needed for workflow verification include:

• case input summary record
• artifact publication record
• artifact check record
• optional structured extraction record when a stage explicitly parses JSON, JSONL, or similar data

The canonical report MUST remain the authoritative fact stream.

Derived summaries and build analytics remain downstream projections.

11. Scope Boundary

This proposal is intentionally local-first.

It optimizes for:

• compilers
• code generators
• corpus processors
• certification pipelines
• lowering chains
• executable handoff validation
• package or artifact verification on one machine

It does not try to solve:

• fleet orchestration
• mutable host configuration management
• arbitrary long-running distributed job control

12. Relationship To Existing Suite Model

This proposal should extend the suite model, not create a separate engine.

The core execution concepts remain:

• cases
• actions
• assertions
• deterministic report order

The main addition is that a case may now be discovered from an explicit workflow source and may carry named artifacts across multiple actions.

In other words:

• unit-style inventory runs are one small special case
• staged workflow verification is the larger model

13. Representative Example

Consider a corpus verification workflow with these stages:

1. discover source cases under a corpus directory
2. compile each case to typed_unit.jsonl
3. certify the produced JSONL
4. lower certified output to SIR
5. lower SIR to executable form
6. run the executable
7. assert expected exit code or stdout

The important property is not the exact syntax.

The important property is the explicit model:

• case key is the corpus fixture name
• compile stage publishes named artifact typed_unit
• certify stage consumes typed_unit
• lower-to-sir stage consumes typed_unit or certification output explicitly
• executable stage publishes named artifact executable
• run stage consumes executable
• assertions reference structured action outcomes and declared expected values

This is the class of workflow that Observer should own so users stop rewriting bespoke Python or shell orchestration.

14. Proposed First Deliverable

The smallest useful workflow milestone should support:

• filesystem-based deterministic case discovery
• ordered multi-action case evaluation
• named artifact publication and lookup
• proc actions over local tools
• file existence and JSON parse checks
• assertions over exit codes, stdout, stderr, and parsed JSON fields
• canonical JSONL reporting for all workflow stages

That milestone would already replace a large class of ad hoc local verification scripts.

15. Design Standard

Observer should only pursue this direction if it maintains the same hard priorities as the rest of the platform:

1. deterministic
2. canonical
3. mechanically derived from explicit contract data

If a workflow feature would fall back to heuristic path guessing, implicit stage coupling, or ambiguous case identity, it should be rejected.