Crash diagnostics

[Joshpolansky]

Add crash diagnostics. Add unhandled exception handlers and create stack and data dumps.

[Copilot]

Pull request overview

Adds a cross-cutting crash diagnostics subsystem to Loom’s runtime: thread-local execution breadcrumbs, exception guarding/quarantining for module entrypoints, persistent fault report storage, and HTTP APIs to surface faults to the UI. It also stamps build identity (SDK version + git SHA + build type) into reports and introduces a dedicated “crasher” module plus unit tests to exercise the diagnostics pipeline.

Changes:

• Introduce loom::diag components: breadcrumb tracking, guarded calls + fault sink, persistent FaultStore, structured FaultReport, and a process-global CrashHandler.
• Surface fault status and reports via runtime server JSON (/api/modules stats additions, plus /api/faults and /api/faults/:id).
• Add build/runtime plumbing: git SHA stamping in SDK, cpptrace dependency for symbolization, and a modules/crasher test module + new unit tests.

Reviewed changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
tests/test_diag.cpp	New unit tests for breadcrumbs, guard(), FaultReport JSON, and FaultStore persistence.
tests/CMakeLists.txt	Adds diag test and links required runtime diag sources into the test binary.
sdk/include/loom/version.h.in	Adds kGitSha build identifier for crash reports.
sdk/include/loom/bus.h	Prevents exceptions from service handlers unwinding into callers by catching and returning an error.
sdk/CMakeLists.txt	Stamps git SHA (and “dirty” state) into generated SDK version header without failing builds when git metadata is absent.
runtime/src/server.cpp	Adds module last-fault fields to module JSON and introduces /api/faults endpoints.
runtime/src/scheduler.cpp	Wraps module lifecycle entrypoints with diag guard and records/report faults (quarantine + sink notification).
runtime/src/runtime_core.cpp	Instantiates FaultStore and wires a RuntimeFaultSink into the scheduler.
runtime/src/run.cpp	Installs process-global crash handler writing reports to <dataDir>/crash.
runtime/src/diag/symbolizer.cpp	Implements address→symbol resolution via cpptrace.
runtime/src/diag/runtime_fault_sink.cpp	Builds/persists/publishes structured fault reports when a module call throws.
runtime/src/diag/fault_store.cpp	Implements persistent fault report registry loaded from <dataDir>/crash.
runtime/src/diag/fault_report.cpp	Implements structured report JSON serialization with embedded JSON sections.
runtime/src/diag/crash_handler.cpp	Implements POSIX/Windows crash capture for fatal signals/SEH and unhandled exceptions.
runtime/src/diag/breadcrumb.cpp	Defines the thread_local breadcrumb storage.
runtime/include/loom/scheduler.h	Extends task state with last-fault diagnostics fields and adds fault sink plumbing.
runtime/include/loom/runtime_core.h	Exposes faultStore() and stores FaultStore + injected sink.
runtime/include/loom/diag/symbolizer.h	Public symbolizer API + SymFrame definition (cpptrace kept out of headers).
runtime/include/loom/diag/runtime_fault_sink.h	Declares runtime-layer implementation of IFaultSink.
runtime/include/loom/diag/guard.h	Adds exception guard wrapper for module calls with breadcrumb attribution.
runtime/include/loom/diag/fault_store.h	Declares persistent fault report store API used by the server.
runtime/include/loom/diag/fault_sink.h	Declares the scheduler→runtime fault sink interface and event payload.
runtime/include/loom/diag/fault_report.h	Declares structured fault report model and JSON serialization API.
runtime/include/loom/diag/crash_handler.h	Declares process-global crash handler install API.
runtime/include/loom/diag/breadcrumb.h	Declares breadcrumb model, phases, and RAII scope helper.
runtime/conanfile.py	Adds cpptrace dependency to runtime package.
runtime/CMakeLists.txt	Adds diag sources, defines build-type macro, and links cpptrace privately.
modules/crasher/crasher.cpp	Adds a deliberately-faulting module to exercise crash diagnostics paths.
modules/crasher/CMakeLists.txt	Builds and installs the new crasher module plugin.
modules/CMakeLists.txt	Includes the new crasher module in the build.
conanfile.py	Adds cpptrace to top-level Conan deps (runtime-only usage).
CMakeLists.txt	Adds find_package(cpptrace REQUIRED) at the project level.
.gitignore	Ignores _crashtest/ output directory.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 33 changed files in this pull request and generated 5 comments.

[Joshpolansky]

Thanks Copilot — addressed all 5 findings from the re-review in a45748b:

1. scheduler.cpp long-running thread UAF — stop() erased the TaskState before joining the threads (join is outside the lock). Now the owning unique_ptr is moved into a local that outlives the join, so the TaskState stays valid until the threads have exited. (stopAll() already joined before clearing.)
2. scheduler.cpp isolatedLoop quarantine — I/O mappings + oscilloscope sampling now skip a faulted module too (gated behind !faulted), matching classLoop.
3. crash_handler.cpp async-signal-safety — dropped the non-async-signal-safe signal() call (SA_RESETHAND already resets the disposition) and re-raise via kill(getpid(), sig) on both the report path and the re-entrant path.
4. runtime_fault_sink.cpp path traversal — the report id (→ <crashDir>/<id>.json) now sanitizes the module portion to a filename-safe set, so a / or .. from the instantiate body can't escape the crash dir.
5. fault_store.cpp record() silent persist failure — a failed file open is now detected and logged. I deliberately keep the in-memory entry so a fault that actually happened still appears in /api/faults for the session (the warning makes the non-persistence explicit) rather than dropping it entirely.

The earlier 7 comments (fcntl/SIGSTKSZ build breaks, strlen signal-safety, missing includes, lastFaultMsg race) were fixed in f995360; macOS + Linux + Windows builds are green there. 114 tests pass.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 33 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

runtime/src/scheduler.cpp:852

• In the class loop, the return value of diag::guard() is ignored. If cyclicGuarded() throws, recordModuleFault() quarantines the module, but this tick will still run oscilloscope_->sampleModule and update timing/cycle counters for the faulting module. That contradicts the intent of skipping faulted modules and may touch module state after an exception.

            // Execute (cyclicGuarded acquires module's runtimeMutex_ so
            // server/watch threads can't race on runtime_ reads)
            diag::guard(diag::Phase::Cyclic, member.moduleId.c_str(), member.mod->className.c_str(),
                        [&]{ member.mod->instance->cyclicGuarded(); },
                        [&](const diag::FaultInfo& f){ faultMember(member, f); });

            // Lightweight sampling: use oscilloscope fast-path.
            // A member in runner.members is guaranteed alive — removeMember() pauses
            // the class and waits for ack BEFORE erasing, so no moduleMutex_ needed.
            // We try_lock the module's runtimeMutex_ so we never block the caller;
            // missing one sample tick is acceptable.
            if (oscilloscope_ && dataEngine_) {
                int64_t nowMs = static_cast<int64_t>(
                    std::chrono::duration_cast<std::chrono::milliseconds>(
                        std::chrono::system_clock::now().time_since_epoch()).count());
                oscilloscope_->sampleModule(member.moduleId, *dataEngine_,
                                            *member.mod->instance, nowMs);
            }

runtime/src/scheduler.cpp:1017

• In isolatedLoop(), after a module is quarantined (state.faulted becomes true), the thread continues ticking forever and still records lastCycleTimeUs/cycleCount/history samples even though no module code runs. This is inconsistent with the class-based loop (which stops incrementing per-module counters once faulted) and can mislead the UI into thinking the module is still executing.

        auto t0 = std::chrono::steady_clock::now();
        if (!state.faulted.load()) {
            diag::guard(diag::Phase::Cyclic, mod.id.c_str(), mod.className.c_str(),
                        [&]{ mod.instance->cyclicGuarded(); },
                        [&](const diag::FaultInfo& f){
                            recordModuleFault(state, mod, f.phase, f.message);
                        });
        }
        auto t1 = std::chrono::steady_clock::now();

        // Once quarantined, skip I/O mappings + sampling too (the class loop skips
        // faulted members entirely) — don't keep touching a module in an error state.
        if (!state.faulted.load()) {
            // Execute I/O mappings for this isolated module
            if (ioMapper_) {
                ioMapper_->executeForModule(mod.id);
            }

            // Sample this isolated module using oscilloscope fast-path.
            // The isolated thread owns this module exclusively; state.running guards lifetime.
            if (oscilloscope_ && dataEngine_) {
                int64_t nowMs = static_cast<int64_t>(
                    std::chrono::duration_cast<std::chrono::milliseconds>(
                        std::chrono::system_clock::now().time_since_epoch()).count());
                oscilloscope_->sampleModule(mod.id, *dataEngine_, *mod.instance, nowMs);
            }
        }

        auto elapsed = std::chrono::duration_cast<std::chrono::microseconds>(t1 - t0);
        state.lastCycleTimeUs.store(elapsed.count());
        state.cycleCount.fetch_add(1);