Skip to content

30-suite.md

Latest commit

 

History

History
584 lines (391 loc) · 21.2 KB

30-suite.md

File metadata and controls

584 lines (391 loc) · 21.2 KB

Suite Model and Language Surfaces v0 Draft

1. Role

A suite defines how cases are discovered and how their outcomes are judged.

Observer has one semantic suite core with two authoring surfaces:

  • simple surface for routine expectations
  • full script surface for advanced testing

Both surfaces MUST lower to the same core model.

Inventory-driven execution is one important case-source form, but it is not the only one.

The suite core MUST also be able to model staged workflow cases discovered from explicit local inputs and evaluated through ordered action plans.

2. Core Concepts

A suite core is composed of items.

Each suite item contains:

  • item id: stable identity within the suite
  • case source: rule that chooses zero or more canonical case keys
  • body: ordered action and assertion logic evaluated once per discovered case
  • selection mode: required or optional

A case identity is the pair:

  • suite item id
  • case key

This case identity is canonical for reporting.

For inventory-driven items, the case key is the selected inventory test name.

For workflow-driven items, the case key is the stable key produced by the declared discovery rule.

3. Case Sources

The suite core supports two families of case sources:

  • inventory selectors
  • workflow discovery sources

Every case source MUST define:

  • the canonical case key
  • the deterministic case order
  • the values made available to the case body

3.1 Inventory Selectors

A selector matches inventory test names.

Initial selector forms in v0:

  • exact name
  • prefix
  • glob
  • regex
  • any-of composite
  • all-of composite

Selector evaluation is performed against canonical inventory names only.

Selected names MUST be processed in canonical inventory order.

For inventory selectors, the selected inventory test name is both:

  • the canonical case key
  • the default bound value for the case body

3.2 Workflow Discovery Sources

Workflow discovery sources enumerate cases from explicit local inputs rather than from inventory test names.

Initial workflow discovery forms are intentionally narrow and deterministic, such as:

  • files under a declared root and pattern
  • rows from a declared manifest file
  • entries from a previously published structured artifact

Each workflow discovery source MUST define:

  • how the case key is derived
  • which case input fields are bound into the body
  • which ordering rule is used

If discovery depends on filesystem enumeration, the resulting cases MUST be ordered by explicit canonical path normalization rules rather than host iteration order.

The first concrete workflow discovery source locked for v0 is filesystem discovery.

That source enumerates files under one declared root using one declared glob and derives the canonical case key from one declared path field.

4. Selection Modes

A suite item is either:

  • required: empty selection is an error
  • optional: empty selection produces zero cases

Required selection is the default.

5. Actions and Assertions

An action observes the system under test and returns either:

  • ok value
  • fail value

An assertion consumes ordinary values or fail values and records pass or fail outcomes.

Observer MUST continue evaluating a case after assertion failure unless a future suite construct explicitly requests early abort.

Bindings are introduced only by:

  • suite item case-source bindings
  • ifOk: success bindings
  • ifFail: failure bindings

There are no general mutable locals in v0.

Workflow-capable suites additionally permit explicit artifact publication and later artifact lookup through named bindings. Those bindings MUST be mechanically derived from prior action results, not inferred from free-form command text.

Artifact bindings are scoped to one case. Re-publishing the same artifact name within one case is a suite runtime error.

6. Simple Surface

The simple surface exists for the common case where the suite author wants to say:

  • select one test or a small group of tests
  • run them through their inventory binding
  • assert a small number of expected outcomes

The exact concrete syntax may remain compact, but its semantics are fixed:

  • simple items always lower to inventory-driven run actions
  • simple expectations are assertions over the canonical run result
  • there is no separate simple execution engine

The simple surface does not define workflow discovery or artifact-aware staged execution in v0. Those capabilities belong to the full script surface and future workflow-specific sugar that still lowers to the same suite core.

6.1 Concrete Simple Syntax

Observer v0 defines this first concrete simple syntax:

test <Selector> [timeoutMs: <u32>] [optional]: expect <SimplePredicate>.

test <Selector> [timeoutMs: <u32>] [optional]: [
	expect <SimplePredicate>.
	expect <SimplePredicate>.
].

Examples:

test "Smoke::Version" timeoutMs: 2000: expect exit = 0.

test prefix: "Smoke::" timeoutMs: 2000: [
	expect exit = 0.
	expect out contains "version".
].

optional marks the suite item as optional selection. If omitted, selection is required.

If timeoutMs: is omitted, the runner MUST apply the implementation default timeout for inventory-driven runs and MUST record that effective timeout in execution metadata.

Normalized suite bytes and canonical item id derivation are defined in 25-normalization.md.

6.2 Simple Selectors

The simple surface supports these selector spellings:

  • "Name" for exact name selection
  • prefix: "Prefix"
  • glob: "Pattern"
  • regex: /Pattern/

These selectors lower directly to the canonical selector model defined by the suite core.

6.3 Simple Predicates

The simple surface supports predicates over the canonical run result only.

Initial predicate forms are:

  • exit = <int>
  • exit != <int>
  • out = <string>
  • err = <string>
  • out contains <string>
  • err contains <string>
  • out match /regex/
  • err match /regex/

The simple surface does not directly expose arbitrary actions. If a test requires protocol probes, branching, or direct process execution, the full script surface MUST be used.

6.4 Lowering Rule

Every simple suite item lowers mechanically to this conceptual core form:

  • select inventory test names using the declared selector
  • for each selected test name, evaluate run with the effective timeout
  • assert each simple predicate against the returned canonical run result

No heuristic inference is permitted during lowering.

A representative simple item is conceptually equivalent to:

  • select tests
  • run each selected test through inventory binding
  • assert exit code, output, error output, or other canonical run fields

7. Full Script Surface

The full script surface exists for cases where correctness depends on explicit observation logic.

The full surface MAY include:

  • inventory-driven run actions
  • workflow case discovery
  • direct process actions
  • explicit artifact publication and lookup
  • file and structured-data observations
  • HTTP or HTTPS observations
  • TCP observations
  • result branching
  • conditional assertions
  • structured grouping

The full surface MUST remain deterministic in evaluation order.

The first workflow-capable concrete syntax defined here is intentionally narrow. It supports deterministic filesystem case discovery, dynamic process arguments derived from case inputs and artifact bindings, explicit artifact publication, artifact checks, and JSON or JSONL extraction.

7.1 File Shape

A full script file MUST be UTF-8 encoded.

A full script file MAY begin with one module header:

module <Identifier>.

After the optional module header, the file MUST contain one or more suite items and no other top-level forms.

Line comments begin with ;; and continue to end of line.

7.2 Suite Item Syntax

The concrete suite item forms are:

(<Selector>) forEach: [ :<Binding> | <Statement>* ].

(<Selector>) forEachOptional: [ :<Binding> | <Statement>* ].

(files: <String> glob: <String> key: <CaseKeyField>) forEachCase: [ :<Binding> | <Statement>* ].

(files: <String> glob: <String> key: <CaseKeyField>) forEachCaseOptional: [ :<Binding> | <Statement>* ].

Rules:

  • forEach: creates a required-selection item.
  • forEachOptional: creates an optional-selection item.
  • <Binding> is the selected inventory test name for each case.
  • statement evaluation order is the exact source order in the block.
  • forEachCase: creates a required workflow-discovery item.
  • forEachCaseOptional: creates an optional workflow-discovery item.
  • in workflow items, <Binding> is a structured case input object with canonical fields described below.

For filesystem discovery items, the bound case object exposes these canonical fields:

  • key: canonical case key string for the discovered file
  • path: normalized path string rooted at the declared discovery root
  • name: basename including extension
  • stem: basename without extension
  • ext: extension without leading dot, or empty string if none

<CaseKeyField> MUST currently be one of:

  • path
  • name
  • stem

7.3 Statement Forms

Permitted statement forms are:

expect: <Predicate>.

publish: <String> kind: <ArtifactKind> path: <ValueExpr>.

(<ResultExpr>) ifOk: [ :<Binding> | <Statement>* ] ifFail: [ :<Binding> | <Statement>* ].

(<BoolExpr>) ifTrue: [ <Statement>* ] ifFalse: [ <Statement>* ].

Rules:

  • ifFail: MAY be omitted; omission means no explicit failure handler.
  • ifFalse: MAY be omitted; omission means no-op.
  • nested blocks do not introduce new suite items.
  • the runner MUST continue executing later statements after a failed assertion.
  • publish: introduces or validates one named artifact binding for the current case.

7.4 Core Expressions

Result-producing expressions:

run: <ValueExpr> timeoutMs: <u32>
proc: <String> args: <ArgArray> timeoutMs: <u32>
httpGet: <String> timeoutMs: <u32>
tcp: <String> send: <BytesExpr> recvMax: <u32> timeoutMs: <u32>
artifactCheck: <String> kind: <ArtifactKind>
extractJson: <String> select: <String>
extractJsonl: <String> select: <String>

Value access and constructors:

(<ValueExpr> <FieldName>)
(<ValueExpr> header: <String>)
artifactPath: <String>
joinPath: <ArgArray>
Fail msg: <String>

Predicate forms:

<ValueExpr> isStatusClass: <Int>
<ValueExpr> isStatus: <Int>
<ValueExpr> hasHeader: <String>
(<ValueExpr>) = <ValueExpr>
(<ValueExpr>) != <ValueExpr>
(<ValueExpr>) < <ValueExpr>
(<ValueExpr>) <= <ValueExpr>
(<ValueExpr>) > <ValueExpr>
(<ValueExpr>) >= <ValueExpr>
(<ValueExpr>) contains: <ValueExpr>
(<ValueExpr>) contains: /<Regex>/
(<ValueExpr>) startsWith: <ValueExpr>
(<ValueExpr>) endsWith: <ValueExpr>
(<ValueExpr>) match: /<Regex>/

Observer v0 does not define user functions, mutation, loops, arbitrary file IO, or implicit shell execution in the script surface.

Workflow support does not relax those restrictions. The goal is explicit staged verification, not general scripting.

7.5 Script Grammar

Script         ::= WS? ModuleHeader? WS? Item+ WS?
ModuleHeader   ::= "module" WS Identifier "."
Item           ::= InventoryItem | WorkflowItem
InventoryItem  ::= "(" Selector ")" WS ForEachKind WS Block "."
WorkflowItem   ::= "(" "files:" WS String WS "glob:" WS String WS "key:" WS CaseKeyField ")" WS WorkflowForEachKind WS Block "."
ForEachKind    ::= "forEach:" | "forEachOptional:"
WorkflowForEachKind ::= "forEachCase:" | "forEachCaseOptional:"
Block          ::= "[" WS? ":" Identifier WS "|" WS? Statement* WS? "]"

Statement      ::= ExpectStmt | PublishStmt | ResultBranchStmt | BoolBranchStmt
ExpectStmt     ::= "expect:" WS Predicate "."
PublishStmt    ::= "publish:" WS String WS "kind:" WS ArtifactKind WS "path:" WS ValueExpr "."
ResultBranchStmt ::= "(" ResultExpr ")" WS "ifOk:" WS Block WS ("ifFail:" WS Block)? "."
BoolBranchStmt ::= "(" BoolExpr ")" WS "ifTrue:" WS PlainBlock WS ("ifFalse:" WS PlainBlock)? "."
PlainBlock     ::= "[" WS? Statement* WS? "]"

ResultExpr     ::= RunExpr | ProcExpr | HttpGetExpr | TcpExpr | ArtifactCheckExpr | ExtractJsonExpr | ExtractJsonlExpr
RunExpr        ::= "run:" WS ValueExpr WS "timeoutMs:" WS UInt
ProcExpr       ::= "proc:" WS String WS "args:" WS ArgArray WS "timeoutMs:" WS UInt
HttpGetExpr    ::= "httpGet:" WS String WS "timeoutMs:" WS UInt
TcpExpr        ::= "tcp:" WS String WS "send:" WS BytesExpr WS "recvMax:" WS UInt WS "timeoutMs:" WS UInt
ArtifactCheckExpr ::= "artifactCheck:" WS String WS "kind:" WS ArtifactKind
ExtractJsonExpr ::= "extractJson:" WS String WS "select:" WS String
ExtractJsonlExpr ::= "extractJsonl:" WS String WS "select:" WS String

Predicate      ::= StatusClassPredicate | StatusPredicate | HeaderPredicate | ComparePredicate | ContainsPredicate | MatchPredicate | PrefixPredicate | SuffixPredicate | FailExpr
StatusClassPredicate ::= ValueExpr WS "isStatusClass:" WS Int
StatusPredicate ::= ValueExpr WS "isStatus:" WS Int
HeaderPredicate ::= ValueExpr WS "hasHeader:" WS String
ComparePredicate ::= ValueExpr WS CompareOp WS ValueExpr
ContainsPredicate ::= "(" ValueExpr ")" WS "contains:" WS (ValueExpr | Regex)
MatchPredicate ::= "(" ValueExpr ")" WS "match:" WS Regex
PrefixPredicate ::= "(" ValueExpr ")" WS "startsWith:" WS ValueExpr
SuffixPredicate ::= "(" ValueExpr ")" WS "endsWith:" WS ValueExpr

BoolExpr       ::= Predicate
ValueExpr      ::= Identifier | String | BytesExpr | Int | FieldAccess | HeaderLookup | ArtifactPathExpr | JoinPathExpr
FieldAccess    ::= "(" ValueExpr WS FieldName ")"
HeaderLookup   ::= "(" ValueExpr WS "header:" WS String ")"
ArtifactPathExpr ::= "artifactPath:" WS String
JoinPathExpr   ::= "joinPath:" WS ArgArray
FailExpr       ::= "Fail" WS "msg:" WS String
BytesExpr      ::= String "#utf8"
ArgArray       ::= "[" WS? (ValueExpr (WS? "," WS? ValueExpr)*)? WS? "]"

ArtifactKind   ::= String
CaseKeyField   ::= "path" | "name" | "stem"

Selector       ::= ExactSelector | PrefixSelector | GlobSelector | RegexSelector | AnySelector | AllSelector
ExactSelector  ::= String
PrefixSelector ::= "prefix:" WS String
GlobSelector   ::= "glob:" WS String
RegexSelector  ::= "regex:" WS Regex | Regex
AnySelector    ::= "any:" WS "[" WS? SelectorList WS? "]"
AllSelector    ::= "all:" WS "[" WS? SelectorList WS? "]"
SelectorList   ::= Selector (WS? "," WS? Selector)*

CompareOp      ::= "=" | "!=" | "<" | "<=" | ">" | ">="
FieldName      ::= Identifier
Identifier     ::= IdentStart IdentContinue*
UInt           ::= Digit+
Int            ::= "-"? Digit+
Regex          ::= "/" { RegexChar } "/"
String         ::= DQuote { Char | Escape } DQuote
WS             ::= (" " | "\t" | "\n" | "\r")+

7.6 Deterministic Validation

The runner MUST reject a full script as a suite syntax error if it contains:

  • unknown top-level forms
  • unsupported selector or action forms
  • malformed regex or byte literals
  • references to unbound identifiers
  • any construct forbidden by this document

8. Canonical Action Set

The semantic core supports these initial actions:

  • run: execute an inventory test by canonical test name
  • proc: execute a concrete process path and argv
  • httpGet: observe an HTTP or HTTPS endpoint
  • tcp: observe a TCP service
  • artifactPublish: publish one named artifact binding
  • artifactCheck: validate a named artifact binding
  • extractJson: extract one bounded value from a JSON artifact
  • extractJsonl: extract one bounded value from a JSONL artifact

The distinction is intentional:

  • run is inventory-driven and canonical for named tests
  • proc is direct process execution and does not accept an inventory test name
  • workflow artifact actions operate over explicit named bindings, not implicit path reconstruction

Workflow suites MUST pass artifact paths into later proc actions explicitly through artifactPath: <String> and explicit args: values. The runner MUST NOT rewrite shell strings or infer output paths from prior command lines.

8.1 Result Model

Every action returns Result[T] with one of two states:

  • ok value of type T
  • fail value with structured diagnostic data

When an action is used in ifFail:, the failure binding MUST expose this canonical fail value directly.

The canonical fail shape is:

  • msg: non-empty string
  • kind: stable failure classifier string
  • code: optional implementation-specific code string

Field access over a fail binding supports kind, msg, and code. Accessing code when no code is present is a suite runtime error.

kind SHOULD use a small stable vocabulary such as timeout, spawn, io, protocol, http, tcp, unknown-test, artifact-missing, or artifact-kind.

Workflow-aware runners SHOULD preserve the distinction between infrastructure problems, workflow stage failures, assertion failures, and invariant failures through stable failure classification.

8.2 Canonical Run Result

The canonical result type for run and proc is ProcRun with fields:

  • exit: signed integer process or provider status code
  • out: raw output bytes
  • err: raw diagnostic bytes

run resolves the selected inventory entry and executes it according to inventory binding:

  • provider entries invoke the provider host using provider id plus provider target
  • exec entries spawn the declared executable path with declared argv
  • sh entries execute only through explicit shell opt-in

If the referenced inventory test name does not exist, run MUST return a fail value with kind: "unknown-test".

If execution exceeds timeoutMs, the action MUST return a fail value with kind: "timeout".

8.3 Other Action Result Shapes

httpGet returns HttpResp with fields:

  • status: integer response status
  • body: raw body bytes
  • headers: map of lowercase string to string

Header names are canonicalized to lowercase ASCII. Header lookup via (<ValueExpr> header: <String>) lowercases the requested name before lookup and returns the matching string value. Missing headers are suite runtime errors.

The raw headers map is part of the canonical result model but is not a first-class script value in v0. The full script surface supports explicit header lookup only; it does not define general map values or general map indexing.

To test optional header presence without converting absence into a runtime error, the full script surface defines <ValueExpr> hasHeader: <String>. This predicate lowercases the requested header name before lookup and returns true if that canonical header is present, otherwise false.

To state HTTP status intent directly, the full script surface defines <ValueExpr> isStatus: <Int>. This predicate is valid only for canonical HTTP responses and returns true when the response status equals the requested integer.

To state HTTP status family intent directly, the full script surface defines <ValueExpr> isStatusClass: <Int>. This predicate is valid only for canonical HTTP responses and returns true when integer division of the response status by 100 equals the requested class. Valid classes are 1 through 5.

tcp returns TcpResp with fields:

  • bytes: raw received bytes

artifactCheck returns ArtifactRef with fields:

  • name: artifact binding name
  • kind: artifact kind string
  • path: normalized artifact path string

extractJson and extractJsonl return ExtractValue with fields:

  • format: json or jsonl
  • select: declared selector string
  • count: number of matched values
  • text: bounded canonical UTF-8 text rendering of the selected value

artifactPath: <String> resolves to the normalized path string of the named artifact binding in the current case. Referencing an unknown artifact name is a suite runtime error.

joinPath: <ArgArray> resolves to one normalized path string by joining its evaluated string operands with the canonical / separator and then normalizing the result. It exists so workflow suites can derive downstream artifact paths explicitly without hidden string interpolation.

All output-carrying actions use raw bytes as the canonical payload representation.

8.4 Bytes and Strings

Observer distinguishes between strings and bytes.

  • string literals are UTF-8 text values
  • byte literals are formed by "..."#utf8
  • action outputs such as out, err, body, and bytes are canonical bytes, not strings

Conversion rules:

  • comparing bytes with a string literal for =, !=, contains:, startsWith:, or endsWith: encodes the string literal as UTF-8 bytes
  • match: and regex-based contains: against bytes decode the byte operand as UTF-8 text
  • if a byte operand is not valid UTF-8, regex operations over that operand MUST evaluate to false rather than raising a script error

Observer MUST NOT silently reinterpret arbitrary bytes using locale-dependent decoding.

8.5 Predicate Domains

Comparison operators <, <=, >, and >= are defined only for integers.

Equality and inequality are defined for:

  • integers
  • booleans
  • strings
  • bytes

The runner MUST raise a suite runtime type error if a predicate is applied to unsupported operand types.

9. Failure Handling

If an action returns fail and the suite item does not explicitly handle that failure, the runner MUST record an unhandled action failure for the case.

A case passes only if:

  • all recorded assertions pass
  • zero unhandled action failures are recorded

10. Design Rule

If a capability can be expressed either as simple syntax or script syntax, both forms MUST lower to the same canonical suite core.

This rule is not optional. It is what keeps Observer coherent as one platform instead of two loosely related testing tools.