Span-Linked Distributed Tracing Context Propagation Through Soroban Cross-Contract Calls

[elizabetheonoja-art]

Problem Statement / Feature Objective

OpenTelemetry distributed tracing currently captures spans within the Rust backend but breaks at the Soroban boundary: each cross-contract call from the settlement pipeline to the Soroban runtime is a black box with no propagated trace context. When a mint_token call fails due to a contract panic or a rollback_batch call exceeds its gas limit, the root cause is invisible in traces — engineers must manually correlate Soroban logs with backend spans. A trace context propagation mechanism must be implemented that injects the current trace_id, span_id, and trace_flags into Soroban contract calls via the contract's env::invoke_contract arguments, and captures Soroban execution spans on the host side via Soroban's diagnostic event emission.

Technical Invariants & Bounds

• Trace context serialized as a 32-byte trace_id + 8-byte span_id + 1-byte trace_flags (W3C traceparent format), hex-encoded into a 73-byte ASCII string.
• Context passed as an additional BytesN<73> argument to every Soroban contract invocation. The contract extracts it and emits a diagnostic_event at each entry/exit point.
• Soroban diagnostic events have a max size of 1,024 bytes per event; the trace context payload plus a 1-byte event type code fits well within this limit.
• Each Soroban invocation may spawn up to 5 nested sub-calls (depth limit per Soroban's max_call_depth config); each sub-call propagates the same trace_id with a new span_id.
• Host-side span capture: a background task polls the Soroban RPC get_events endpoint every 500 ms, filters for diagnostic events matching the known trace context pattern, and exports them as OpenTelemetry spans.
• Trace context injection must add ≤ 100 µs overhead per contract call on the Rust side (negligible compared to the ~500 ms Soroban round-trip).

Codebase Navigation Guide

• src/tracing/soroban_propagator.rs — new module: context injection into Soroban invocations and extraction from diagnostic events.
• src/blockchain/soroban/client.rs — extend all contract invocation methods to accept an optional trace_context: &TraceContext parameter.
• src/blockchain/soroban/contracts/settlement.wat — add WAT code that extracts the trace context from the extra argument and emits diagnostic events at function entry/exit.
• src/tracing/exporters/soroban.rs — new module: OpenTelemetry span exporter that reads Soroban diagnostic events.
• src/tracing/lib.rs — create SorobanTraceLayer that wraps the Soroban client and automatically injects the current span context.
• src/config/default.toml — add [tracing.soroban_propagation] with { enabled = true, poll_interval_ms = 500 }.
• tests/tracing/soroban_trace_propagation_test.rs — integration test using a local Soroban devnet.

Implementation Blueprint

1. In soroban_propagator.rs, define fn inject_context(trace_context: &SpanContext) -> String that hex-encodes trace_id, span_id, trace_flags into a 73-byte ASCII string, and fn extract_context(diagnostic_event: &SorobanEvent) -> Option<(TraceId, SpanId, TraceFlags)> that parses it back.
2. In client.rs, add an overload for each contract method (e.g., async fn submit_batch_proof(&self, ..., trace_ctx: Option<TraceContext>)) that prepends the trace context as an additional ScVal::Symbol argument to the Soroban invoke_contract operation. If trace_ctx is None, pass an empty placeholder.
3. In the Soroban contract (WAT), at the top of each exported function, emit a diagnostic_event with type 0x01 (entry) containing the trace context bytes. Before returning, emit 0x02 (exit) with the same trace context and an i64 status code. The contract must accept an extra parameter trace_ctx at index 0 and pass it through to sub-calls.
4. In exporters/soroban.rs, implement SorobanEventPoller that runs in a background tokio task: every poll_interval_ms, call get_events on the Soroban RPC, filtering for events where event_type == "diagnostic" and the payload starts with known marker bytes. Parse each event into an OpenTelemetry span skeleton, linking parent-child spans via the span_id chain.
5. In lib.rs, define SorobanTraceLayer that implements tower::Layer for the Soroban client service. On each request, if there is an active OpenTelemetry span, call inject_context; on response, await the poller to report the corresponding Soroban-side span (correlated by the span_id the backend generated). Export both sides as a single composite trace.
6. Add a SpanLink in the OpenTelemetry trace to correlate the backend's HTTP POST /invoke span with the Soroban execution span, using SpanKind::CLIENT for the backend and SpanKind::SERVER for the contract.
7. Write an integration test that calls a Soroban contract with trace propagation, captures both sides of the trace, and verifies the trace_id is identical, the parent span_id of the Soroban span matches the backend span's span_id, and duration deltas are within 1% of the measured wall-clock time.

[clintjeff2]

Hello, I have 6 years of experience working on tasks like this and handling similar challenges across backend, frontend, and smart-contract development. I have carefully read and fully understood the task requirements, and I am confident I can deliver a clean, efficient, and well-tested solution promptly. I always ensure high-quality implementation, maintainability, and adherence to best practices.

Pull request in 5 hours.

[real-venus]

Hey, I've read through the issue and I understand it.

Could you assign it to me? I can finish this within 24 hrs.

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox as part of the Official Campaign campaign!

@clintjeff2's PR #90 was approved and merged by @elizabetheonoja-art.

🏆 @clintjeff2: You earned 45 FoxPoints for this contribution! Your current tier: Specialist (3,551 total points). Track your full progress on GrantFox.

👏 Great work, @clintjeff2! Keep contributing to Utility-Protocol.