TTCN-3 test-execution runtime (interpreter + ntt exec) and ETSI conformance gate#776
Open
rafael2knokia wants to merge 19 commits into
Open
TTCN-3 test-execution runtime (interpreter + ntt exec) and ETSI conformance gate#776rafael2knokia wants to merge 19 commits into
rafael2knokia wants to merge 19 commits into
Conversation
rafael2knokia
added a commit
that referenced
this pull request
Jun 23, 2026
A shared, async-collaboration starting point (for Matthias + upstream ntt, agents welcome) to align the branch with the rest of ntt: - Inventory: the branch is ~99.6% additive (122,746 ins / 488 del, 437 files) - it extends the parser/LSP and adds interpreter/runtime/exec/ codec/port/conformance, not a rewrite. Current state: 97.22% conformance, PR #776 mergeable. - Hard-constraint assessment (evidence-backed): - LSP for ETSI users: extended, not changed - low risk. - Go-to-definition <100ms: off the heavy path (LookupWithDB, not Analyze); position lookup ~25-30us/op, 0 allocs. - Diagnostics: full 156-check Analyze per change is the perf watch-item on huge files - to benchmark/debounce. - ntt run / internal-tool coexistence: run.go + exec.go both exist; new rejections from 156 checks - to reconcile. - Open questions, goals/checklist, and the async-collaboration + gating conventions. No code change. %INT_NO_SW_CHANGE %AI=CLAUDE Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rmance gate
Grow ntt from a TTCN-3 front-end (parser / LSP / tooling) into a full
TTCN-3 test-execution toolchain in Go: a tree-walking interpreter and an
`ntt exec` runtime that compile, execute and verdict TTCN-3 test suites,
validated against the ETSI TTCN-3 conformance suite (4788/4948, 97.22%).
- interpreter/ tree-walking eval: testcases, templates, alt/interleave,
ports (message + procedure), timers, components, OO classes
- runtime/, runtime/exec/ testcase executor, verdicts, port/queue model, --cfg,
reporting, C-ABI/cgo bridge and a pure-Go (goport) port binding
- runtime/codec/ JSON, XML/XER, RAW encvalue/decvalue
- ttcn3/semantic/ additional static checks surfaced via `ntt check`
- ttcn3/syntax/ parser extensions (classes, new node kinds)
- internal/lsp/ rename, signature help, workspace symbols, type definition,
organize imports, folding, references (Go-to-def off the
heavy path; position lookup is microseconds)
- conformance.go ETSI suite harness gating regressions vs a committed baseline
- docs/ conformance roadmap, Go-only embedding, C/C++ test ports
Additive by design (go.mod/go.sum unchanged from master); the execution
engine is not on the front-end hot path unless explicitly invoked. This
branch is squashed for now; a reviewable per-subsystem commit split is
planned as a follow-up.
An unqualified getreply/catch inside a blocking `call(S,...) { }`
response block must treat only S's reply/exception (ETSI 22.3.1 h).
The loopback model previously matched any queued reply/exception, so a
reply/exception left over from an earlier unhandled call on the same
port wrongly matched the block's guard.
- record the signature identifier on every call/reply/raise envelope
(procSignatureName -> PortMessage.Signature)
- stash the blocking call's signature on the scope for its response
block (procCallSigKey, save/restore so nested call blocks compose)
- skip a queued head whose known signature differs from the block's,
in commGuardMatches (bare `[] p.getreply` guard) and in
evalPortReceiveInfo (the `from`-qualified variant)
Scoped to the implicit call-block qualifier: explicit-signature and
getcall matching stay lenient, so the Sem_2204 check fixtures are
untouched. Fixes Sem_220301_CallOperation_019/020; conformance
4788 -> 4790 (97.22% -> 97.26%), 0 per-file regressions. Guarded by
interpreter/proc_signature_test.go.
Off by default, so the ETSI conformance suite is byte-identical
(4790/4948, verified 0 per-file regressions). Callers opt in via
interpreter.TestcaseOptions{RealScheduler: true}; only an external load
driver that embeds ntt to run TTCN-3-side concurrency needs it.
With RealScheduler on, N `alive` PTCs each running
`while(true){ p.send; alt{ [] p.receive [] t_guard.timeout }; t_pace.start; t_pace.timeout }`
run on real goroutines and generate concurrent traffic through their
mapped Go test ports -- the idiomatic TTCN-3 load pattern, which the
default skip/virtual model turns into a no-op (zero requests issued).
- flag: RealScheduler on TestcaseOptions, carried onto TestcaseExec
(realScheduler/mtcID, mu-guarded), read via FindTestcaseExec.
- scheduler: `alive` starts bypass the skip heuristic and fork onto a
goroutine even with while(true)+timer-guard alts; the alt scheduler
gains a combined park (waitForAltCombined) that wakes on port traffic
OR the soonest timer deadline OR the PTC's stop.
- per-PTC stop: `all component.stop` now cancels worker goroutines
(StopPTC + unmap); evalBlockStmts and waitForTimerTimeout honour a
per-PTC stop so a while(true) worker unwinds promptly.
- port routing: TestcaseExec.PortKey qualifies a PTC's port by
component id ("\x00c<id>/<name>"), so N same-named ports get private
instances/queues/drivers and a goport Inject routes each reply back to
the worker that sent it. PortDriver resolves the port type via the
bare name. goport clears its per-instance cache at each testcase
teardown (RegisterExecTeardownHook) so restarting component ids can't
alias a stale TestPort across runs.
Tests (interpreter/realsched_test.go, incl. -race): single-worker loop,
default-off skip (0 sends), 4 workers each receiving their own replies
(4 distinct instances), and cross-testcase instance isolation.
The 97.26% match rate conflates genuinely different outcomes: real execution verdict matches, static parse/semantic rejections, parse-only acceptances (noexecution / no testcase), and negative tests the interpreter merely aborted on. Nothing recorded which was which, so "semantically correct by execution" was invisible. Add ConformanceResult.Provenance (executed | exec-reject | static | parse | parse-only | runtime-error | timeout), set at each classification site, and surface a ConformanceSummary.Provenance histogram plus RealExecRate = executed-matches / considered. The existing PassRate and the --baseline gate are unchanged, so nothing regresses; the new number is purely additive. On the current suite the 97.26% match rate decomposes as 2489 executed (50.54% real execution) + 1764 static + 135 parse + 207 parse-only + 195 exec-reject. Also drop the stale, unreferenced root conformance-baseline.json (a Jun-1 real-execution-only snapshot at 50.98%, now superseded by the live RealExecRate); CI gates on testdata/conformance-baseline.json.
Introduce runtime.SemanticsProfile (Approximate|Strict) as the single coherent carrier the semantic-correctness work converges on; the shipped RealScheduler bool folds into it (RealScheduler:true selects Strict, and TestcaseExec.RealScheduler() now means "strict profile active"). Real concurrent PTC execution is simply the first domain of Strict; further correct-semantics domains attach to the same profile as they land, and the transitional flags are removed once a domain reaches parity. Add `ntt conformance --profile approximate|strict` and `--differential`: the latter re-runs each executed testcase under the strict profile and reports verdict divergences from the approximate (gate) run - the work-list showing exactly where strict semantics change behaviour. The default gate stays approximate, so nothing regresses (97.26% match, 50.54% real execution unchanged).
…tic) First Phase-1 increment toward faithful alt semantics (ES 201 873-4 clause 20). Under ProfileStrict, `alt` now runs evalAltStmtStrict: it keeps the correct part of the best-effort path -- source-order, first-match-wins guard evaluation (a matching guard consumes only its own selected event), [else], repeat, activated defaults -- but REPLACES the verdict-preferring heuristic with honest blocking: when no guard matches it parks on the alt's event sources (port traffic, soonest timer deadline, component transition, or this PTC's stop) and re-snapshots, never fabricating a verdict for a branch whose guard did not fire. Reached only under ProfileStrict (interleave still uses best-effort); the default approximate path and the conformance gate are byte-identical (97.26%, 0 regressions), and the RealScheduler tests now exercise the strict evaluator (all green under -race). Next sub-task, surfaced by --differential: connection-topology routing under strict. Today PortKey namespaces a send to the SENDER's component queue, so a `connect(mtc:p, peer:p)` loopback send doesn't reach the connected peer's queue and the receiver blocks (6 alt-2002 fixtures diverge pass->timeout under strict). Routing connected sends to the peer's qualified queue via the connect graph is the next increment.
Second Phase-1 increment. Under ProfileStrict a send on a CONNECTED port
is now delivered to the connected peer(s)' queue(s) via the connect
graph (new ConnectedPeers + PortKeyFor), tagged with the actual sending
component so the receiver's `from` matches -- instead of the sender's
own component-qualified queue. Falls through to self-delivery when the
port has no peer (loopback-to-self / self-connect).
This fixes the connected-component alt fixtures the differential harness
flagged: e.g. `connect(mtc:p, peer:p); peer.start(fsend()); alt { []
p.receive ... }` now delivers the peer's send to the MTC's queue and the
strict alt matches it (was pass->timeout under strict). On alt-2002 the
strict-vs-approximate divergences collapse to a single pre-existing
NegSem miss unrelated to alt (the rest were strict correctly waiting
real 5s timers, cut off only by a short differential --timeout).
The approximate path and the conformance gate are byte-identical
(97.26%). New guards in interpreter/strict_alt_test.go (connected-peer
receive; timer guard actually fires). -race clean.
Add TestcaseOptions.Context: on cancellation RunTestcaseWith stops the executor (exec.Stop), unwinding a blocked strict alt / timer wait via a bounded watcher goroutine (closed on return, so it never outlives the run). The conformance harness's execVerdict now passes its per-testcase timeout context, so a strict run that blocks forever (honest alt semantics on a stuck fixture) is cancelled on timeout instead of leaking a spinning goroutine after we record "timeout". This unblocks a clean full-suite strict differential: the comm-chapter strict sweep now finishes in bounded wall time (~24s / ~48MB RSS). The default approximate path and the conformance gate are byte-identical (97.26%, 0 regressions); Context is nil for RunTestcase. Guard: TestStrictAlt_ContextCancelsBlockedAlt (-race clean).
Fourth Phase-1 increment, so the strict differential measures real divergences instead of real-timer artifacts. Add TestcaseOptions.DeterministicClock (strict-only, orthogonal to Profile): timers advance the per-testcase virtual clock to their deadline and fire instantly instead of sleeping real wall-clock time. timerExpired reads the virtual clock under the flag; the alt block step advances to the SOONEST timer deadline (nextAltTimerVirtualDeadline) so multi-timer ordering holds (guard evaluation no longer advances the clock in this mode). The conformance harness enables it for strict runs; the default gate (approximate) keeps the real clock and is byte-identical (97.26%). A real load driver leaves it off so timers pace real I/O. Effect: the alt-2002 strict differential drops 6->1 divergences at a 2s budget (timer artifacts vanish; the last is a pre-existing NegSem miss), and the 88 communication-chapter divergences are confirmed genuine (unchanged at 2s vs 5s) -- the real proc-comm/send work-list. Guards (interpreter/strict_alt_test.go, -race): a 30s timer fires instantly; the soonest of two timers wins regardless of clause order.
Extend strict connection-topology routing (previously message-send only) to procedure-based communication. call/reply/raise now deliver to the connected peer(s)' queue(s) via the connect graph (shared helpers strictConnectedTargets / enqueueEnvelopeRouted), so a caller's call reaches the server and the server's reply/raise reaches the caller, instead of landing on the sender's own component-qualified queue. On the communication chapter the strict differential drops 88->63 divergences (pass->fail 28->10). The default (approximate) gate is byte-identical (97.26%), full -race suite green. Guard: TestStrictProc_ConnectedCallReply. Remaining comm divergences are separate sub-clusters (skip/deferred- responder timing for single-port servers, port-array any-from routing, blocking-call semantics) tracked for follow-up.
Extend the "finite responder runs at start when a call is already queued" exception to single-port `getcall; reply/raise` servers under the strict profile. The approximate exception (startBodyIsFiniteResponder) only covered indexed-port responders, so a single-port server on a non-alive PTC stayed skipped and its caller's getreply/catch never matched under strict. Comm-chapter strict differential 63->49. Default (approximate) gate is byte-identical (97.26%), full -race suite green. Guard: TestStrictProc_ConnectedCallReply now uses a plain non-alive `create`.
Under the strict profile, procedure receives (getcall/getreply/catch,
including inside check) now match the signature parameter record and
value/exception template via procReceiveMatches, instead of the lenient
"any envelope of this kind" short-circuit. So e.g.
check(getreply(S:{p:=(100..200)} value ?)) no longer matches a reply
whose parameter is out of range (2204 check fixtures 057/058/081/082).
Comm-chapter strict differential 49->45, and the actively-wrong
pass->fail 2204 divergences drop 12->4; verified 0 fixtures regress
(none goes from correct to diverging). The default (approximate) gate
keeps the lenient match and is byte-identical (97.26%); -race green.
Guard: TestStrictProc_CheckHonoursTemplate.
connect(self:p[i], v:p[i]) records the endpoint under the array BASE name
("p") because resolvePortEndpoint/portRefName drops the `[i]`, while comm
(send/call/reply) uses the indexed name "p[i]". Under approximate this is
harmless (shared name-keyed queues); under strict the connect-graph
lookup missed and the call/reply self-delivered, so a caller's
`any from p.getreply` never matched.
Fix strictConnectedTargets (strict-only): when the exact indexed
endpoint has no peer, fall back to the base-name endpoint and re-apply
the element suffix to the peer's port (splitPortIndex). Comm-chapter
strict differential 45->35, 0 regressions (set-diff verified). Default
gate byte-identical (97.26%), -race green. Guard:
TestStrictProc_PortArrayConnectedRouting.
…trict
Under the strict semantics profile, a non-alive PTC started with a body
that blocks on inter-component procedure communication now runs on a real
goroutine instead of being skipped by the synchronous loopback model. This
lets the two-PTC blocking-call shape execute: a server component blocks in
`getcall` while a client component issues a blocking `call{...}`, its reply
routed back over the connection.
- startBodyBlocksOnComm: fork a started non-alive PTC only when its body
blocks on a `call{}` (CallStmt) or a `getcall`, and only when no call is
already queued (the call-before-start case stays on the inline
finite-responder path). Bodies with an `[else]` clause (finite) or a
`@decoded` redirect (codec decoding not yet implemented) are left on the
skip path. Standalone `getreply`/`receive` bodies are not forked on
their own, since their counterpart (a nowait caller or a send-only
sender) is not forked either.
- deterministicClockEnabled: fall back to the real clock while concurrent
PTCs are live. The virtual clock is only sound single-threaded; with
concurrent PTCs, inter-component events already flow in real time
(fast), and a virtual-clock advance in one goroutine would race a safety
timer in another.
- ComponentRef.{done,alive,verdict} are now private and guarded by a
mutex (IsDone/SetDone/IsAlive/SetAlive/GetVerdict, locked MergeVerdict):
a forked PTC updates its completion/verdict on its own goroutine while
the parent observes them via comp.done / comp.alive / comp.running.
Default (approximate) execution is unchanged; the conformance gate stays
at 97.26%. Strict differential across core_language: 73 -> 54 divergences
(19 fixed, 0 regressions). Race detector clean.
A strict alt clause `[expr] op {...}` is now eligible in a snapshot round
only when its boolean guard `expr` holds (ETSI ES 201 873-4 §20.2). The
snapshot gates on a concretely-false boolean; an Undefined/unmodelled or
non-boolean guard falls through to the communication match, preserving the
prior behaviour where the guard was ignored entirely.
This lets a snapshot discriminate clauses that differ only by their guard
across `repeat` rounds. Default (approximate) execution is unchanged; the
conformance gate stays at 97.26%. Strict differential across core_language
gains 2 fixtures with no regressions.
Adds quiesceScheduler, a discrete-event "quiescence barrier" for the strict interpreter's deterministic clock. It models the same rule Go's testing/synctest applies in the runtime: virtual time advances only when every live participant (the MTC and all live PTC goroutines) is parked, and then jumps to the soonest registered timer deadline, waking the goroutine(s) whose timer fires; a communication event or a peer finishing wakes parked participants at the current instant without advancing time. This is the correct basis for making concurrent timer-driven execution fast (no real sleeps), sound (a safety timer can never fire before an earlier event), and free of the current 2ms-polling backstop's load-dependent flakiness — replacing the ad-hoc mix of a real clock, a virtual clock, and a tick counter selected per call site. The type is self-contained (owns its virtual clock, takes explicit goroutine ids) and fully unit-tested under -race: instant single-timer fire, soonest-timer-wins, comm-wakes-without-advance, terminal deadlock detection, stop, done-waiter wakeups, and a producer/consumer stress. It is not yet referenced by the interpreter; wiring the interpreter's block points onto it is a separate, gated step, so this commit changes no execution behaviour and the conformance gate is unaffected.
…direct fixes) Wires the discrete-event quiescence scheduler (runtime/scheduler.go) into the strict interpreter behind a new DeterministicScheduler option: - runtime: TestcaseExec grows a *quiesceScheduler; VirtualClock/ AdvanceVirtualClock delegate to it when active; SchedGoLive/SchedGoDone/ SchedSignal/SchedPark bridge the interpreter to the barrier; FinishPTC deregisters a finished PTC and signalMessageReady broadcasts a wake. - interpreter: every strict block point parks through the scheduler when active — the alt wait (blockForAltEvents), the bare `T.timeout` (both selector and method forms), and the timer-array `any/all timer.timeout` wait. Timer expiry reads the virtual clock via useVirtualClock. The PTC fork registers with the scheduler (SchedGoLive) and the 50 ms start barrier is skipped (the scheduler orders startup). Validated directly by TestSched_TwoPTCBlockingCall: a two-PTC blocking call with a boolean guard, a return value, and a catch(timeout) fail branch runs to a correct pass at virtual time 0 — the 30 s server timer and 5 s call timer never really elapse. The option is left OFF in the conformance harness for now: enabling it routes blocking-call response blocks through the strict alt, which exposes two pre-existing procedure-redirect gaps the approximate path masked — the getcall `-> param` redirect is dropped in the RedirectExpr evaluator, and positional `param(a,-,b)` binding assigns the whole payload rather than per-field. Fixing those is the next step to turning the scheduler on. Default execution is unchanged; the gate stays at 97.26% and -race is clean.
…rict) Three related procedure-based communication fixes under the strict profile, all gate-safe (default execution unchanged, conformance gate stays 97.26%): - getcall redirect routing: the RedirectExpr evaluator handled receive/trigger/getreply/catch/check but not getcall, so a `p.getcall(...) -> param(...)` redirect was silently dropped. Route getcall through evalPortReceiveInfo like the other receive ops. - positional param binding: a `-> param(v1, -, v3)` redirect bound every target to the whole parameter record instead of the per-position field. Carry the signature declaration in TypeDesc and bind positional targets onto the record's fields in formal-parameter order (signatureParamNames + applyParamRedirect); the named `x := field` form and the single-parameter scalar case are unchanged. - multi-client broadcast: a blocking-`call` client PTC was skipped when another component already had a queued call, because the forkStrict gate used the global HasPendingCalls flag. A client produces a call rather than consuming one, so it must always fork; only getcall responders gate on HasPendingCalls. This unhangs `reply ... to all component` fixtures with two or more clients. Together these make the blocking-call two-PTC fixtures genuinely correct (previously the approximate verdict-preferring heuristic fabricated their passes): Sem_220301_CallOperation_001/002/003/007, Sem_220302_GetcallOperation_001/002/005, and the multi-client Sem_220303_ReplyOperation_001/002. Strict differential across core_language shows no regressions; -race clean. Guards: TestStrictProc_GetcallPositionalParamRedirect, _MultiClientBroadcast.
…ive scheduler Replaces the broadcast quiescence scheduler with coopScheduler, a single-runner "token" cooperative scheduler: exactly one component participant (MTC or PTC) executes at a time, and the token is handed to the next participant in a deterministic order (lowest component id first). Virtual time still advances only at quiescence, to the globally-soonest timer deadline — but interleaving is now deterministic too. This removes the residual goroutine-interleaving nondeterminism of the plain quiescence model: with real goroutines running concurrently between park points, two components' snapshots could race (a client's getreply vs. a server's reply), making blocking-call verdicts flaky run-to-run. Under the token model a client blocking call is reproducibly pass (verified: Sem_220301_CallOperation_001 goes from flaky to 8/8 pass under the scheduler). Wiring: the scheduler is keyed on component id (deterministic, unlike a goroutine id); a forked PTC registers via SchedGoLive(id) and takes the token via SchedAcquireToken(id) at goroutine start; SchedPark(id, ...), SchedGoDone(id) and SchedSignal complete the model. Self-contained and unit-tested under -race: instant single-timer, deterministic token handoff, soonest-timer-wins, terminal deadlock, stop, and a mutual-exclusion stress proving the single-runner invariant. Still gated OFF in the conformance harness: enabling it needs a few procedure-comm fixtures resolved first (the call `catch(timeout)` timer ordering, the nowait-call + separate-alt shape, and any-port getcall). Default (approximate) execution is unchanged; the gate stays at 97.26% and -race is clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch grows
nttfrom a TTCN-3 front-end (parser / LSP / tooling)into a full TTCN-3 test-execution toolchain in Go: a tree-walking
interpreter and an
ntt execruntime that can compile, execute andverdict real TTCN-3 test suites, validated against the ETSI TTCN-3
conformance suite.
ETSI conformance: 4788 / 4948 matched (97.22%), tracked as a CI-style
regression gate.
What's included
interpreter/) — tree-walking evaluation of modules,testcases, functions/altsteps, templates,
alt/interleave, ports(message and procedure-based communication), timers, components, and
TTCN-3 object orientation (classes, inheritance, nested classes).
runtime/,runtime/exec/) — testcaseexecutor, verdict handling, port/queue model, config (
--cfg,[MODULE_PARAMETERS]), reporting, and a C ABI / cgo bridge so C/C++test ports can be driven from the Go runtime.
runtime/codec/) — JSON and XML/XER encode/decode pathsplus RAW
encvalue/decvalueround-tripping.ttcn3/semantic/) — additional static checks(attributes, parametrization, restrictions, type rules, …) surfaced
through
ntt check.conformance.go) — runs the ETSI suite,classifies each file's outcome against its
@verdictannotation, andgates regressions against a committed baseline
(
testdata/conformance-baseline.json).Testing
--regress 0.5against the baseline,with a per-file diff requiring zero regressions for every change.
(17 pass / 0 fail / 2 expected inconc) across changes.
go test ./...for the touched packages, including dedicated unittests for the structural-type-compatibility coercion.
Status & remaining work
97.22% is the practical zero-regression ceiling for incremental work; the
remaining ~137 misses are documented with a per-cluster triage in
docs/conformance/remaining-work.md.They need dedicated deep features (concurrent multi-PTC alt scheduling,
union-alternative tracking, a real RAW codec, procedure-signature
qualification) or are contradictory / mislabeled suite fixtures left
intentionally as-is. The miss inventory and per-slice history live
alongside it under
docs/conformance/.