OpenTelemetry contextvars leak across asynchronous generator boundaries in BaseAgent and Runner

[chriskinzel]

Description

In Python asyncio, OpenTelemetry's default context propagation (tracer.start_as_current_span) relies on contextvars. In BaseAgent.run_async and Runner._run_with_trace, OpenTelemetry spans are created around asynchronous generator blocks (async for event in agen: yield event).

When multiple sub-agent coroutines yield and resume concurrently inside an asyncio.TaskGroup, contextvars can leak across coroutine boundaries if the OpenTelemetry span context is not explicitly detached before yielding. This causes child spans from one worker task to be incorrectly parented to the active span of a concurrent sibling task.

Proposed Solution

Explicitly manage OpenTelemetry context attachment and detachment around generator yield boundaries using context.attach and context.detach:

from opentelemetry import context, trace

span = tracer.start_span("invocation")
token = context.attach(trace.set_span_in_context(span))
try:
    # ...
    async for event in agen:
        context.detach(token)
        try:
            yield event
        finally:
            token = context.attach(trace.set_span_in_context(span))
finally:
    context.detach(token)
    span.end()

[surajksharma07]

@chriskinzel explicitly managing context.attach/detach around yield boundaries is the right fix here.

One thing to validate though: context.detach() can throw ValueError when a task is cancelled mid-execution, which is actually why a similar approach was rolled back previously.

Could you confirm that #5725 wraps those detach calls in try/except ValueError and test it under both the parallel execution and task cancellation scenarios?

A thorough test pass before raising the PR for review would be ideal!