diff --git a/AGENTS.md b/AGENTS.md index 52ce6361..ef818f6a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -156,7 +156,7 @@ The client manages a singleton connection to LaunchDarkly and the associated tel |---|---| | `init_client(options?)` | Auto-discovers and initializes `launchdarkly-server-sdk` (optional dep, loaded via `importlib`). Optional — the first AI API call triggers lazy init when `LD_SDK_KEY` is set. Accepts optional overrides for SDK key, base URIs, service name, environment, and OTLP endpoint. Returns `Awaitable[LDClientInterface]`. On every successful path, including the already-initialized path, flushes `$ld:ai:sdk:info` for any LaunchDarkly AI packages that have not yet reported. Does not take behavior defaults. | | `init_client(client=...)` | **BYOC overload** — accepts a pre-initialized `LDClientInterface`. Stores it directly without calling the SDK. Flushes pending `$ld:ai:sdk:info` events. | -| `configure(*, options)` | Sets process-wide behavior defaults on the lifecycle singleton. `options` is a frozen `SDKOptions` (`capture_content: bool = False`). Keyword-only. Handlers read it at invocation. An explicit factory or wrapper `capture_content` overrides it. Replaceable until the first AI invocation; an identical value is allowed after. Not read from request data. | +| `configure(*, options)` | Sets process-wide behavior defaults on the lifecycle singleton. `options` is a frozen `SDKOptions` (`capture_content: bool = True`). Keyword-only. Handlers read it at invocation. An explicit factory or wrapper `capture_content` overrides it, and an explicit `False` strips content everywhere, including offline evaluation spans. Replaceable until the first AI invocation; an identical value is allowed after. Not read from request data. | | `get_client()` | Returns the initialized `LDClientInterface`. Throws if initialization has not completed. | | `shutdown()` | Flushes all pending events and telemetry, then closes the client. Must be awaited before the process exits. Clears sdk-info reporting so a later client reports again. Restores the built-in SDK defaults. | | `register_ai_sdk_package(name, version)` | Records a LaunchDarkly AI package identity. Handler and convenience packages call this at import time. | @@ -571,14 +571,15 @@ beside the input count, so its handlers add it in. OpenAI and LangChain already input, so theirs pass the figure through. Centralising that rule would double-count for two providers out of three. `SpanUsage` is the type that means the folding is already done. -#### Content is off by default +#### Content is on by default Every factory takes `capture_content: bool | None = None`. `None` reads `configure()` at -invocation, which is off until a process sets `SDKOptions(capture_content=True)`. An explicit -`True` or `False` overrides that default. Guard at the call site as well as inside the helper: the -helper's guard makes a forgotten call site harmless, and the call site's guard avoids serialising -JSON that would then be discarded, once per turn, in a loop. Resolve the bool when the handler -runs, not when it is constructed. +invocation. The built-in default is on until a process sets `SDKOptions(capture_content=False)`. +An explicit `True` or `False` overrides that default, and an explicit `False` strips content +everywhere, including offline evaluation spans. Guard at the call site as well as inside the +helper: the helper's guard makes a forgotten call site harmless, and the call site's guard avoids +serialising JSON that would then be discarded, once per turn, in a loop. Resolve the bool when the +handler runs, not when it is constructed. #### Finish reasons have three mechanisms, not one diff --git a/README.md b/README.md index c66ff29a..216c5464 100644 --- a/README.md +++ b/README.md @@ -542,20 +542,23 @@ This matters most on Anthropic, which reports cache reads and writes beside the than inside it. A turn that reads 19,971 tokens from cache and writes 3,580 more reports an input of 3 from the provider; the span reports 23,554. -### Conversation content is off by default +### Conversation content is on by default -Prompts, model output, tool arguments and tool results are personal data, so a span carries only -metadata unless you ask for more: models, token counts, timings, tool names. +Prompts, model output, tool arguments and tool results are personal data. The built-in default +sends that text. A span carries only metadata — models, token counts, timings, tool names — when +capture is turned off. -Pass `capture_content=True` to a handler factory to include the conversation, or turn it on for every handler from process startup. See [Process-wide defaults](packages/client/README.md#process-wide-defaults). +Pass `capture_content=False` to a handler factory to leave the conversation off that handler, or +turn it off for every handler from process startup. An explicit `False` wins everywhere, including +on offline evaluation spans. See [Process-wide defaults](packages/client/README.md#process-wide-defaults). ```python from launchdarkly_ai_claude_messages import create_claude_messages_handler -handler = create_claude_messages_handler(capture_content=True) +handler = create_claude_messages_handler(capture_content=False) ``` -Turning this on sends the text of every request and response to whatever collector the SDK points +Leaving this on sends the text of every request and response to whatever collector the SDK points at. ### Streaming diff --git a/TELEMETRY-CONTRACT.md b/TELEMETRY-CONTRACT.md index 1016d8f3..6338a82f 100644 --- a/TELEMETRY-CONTRACT.md +++ b/TELEMETRY-CONTRACT.md @@ -397,12 +397,13 @@ error for an abandoned stream, so two dashboards would describe the same run dif Every handler factory takes `capture_content: bool | None = None`. `None` reads the process-wide default at invocation (`configure(options=SDKOptions(...))`), which -is off. `True` or `False` overrides that default for the handler, and an explicit `False` wins -over a configured `True`. A convenience wrapper's `capture_content` is the per-call override: when -the caller passes it, that value is what the handler is built with; when they omit it, the wrapper -passes `None` and the process default applies. Conversation content is personal data. A run emits -only metadata, meaning models, token counts, timings and tool names, until capture is turned on. -Turning it on sends the text of every request and response to whatever collector the SDK points at. +is on. `True` or `False` overrides that default for the handler, and an explicit `False` wins +over a configured `True`, including on offline evaluation spans. A convenience wrapper's +`capture_content` is the per-call override: when the caller passes it, that value is what the +handler is built with; when they omit it, the wrapper passes `None` and the process default +applies. Conversation content is personal data. The default sends the text of every request and +response to whatever collector the SDK points at. An explicit `False` emits only metadata, meaning +models, token counts, timings and tool names. Guard twice, on purpose. The helper returns without writing when capture is off, which makes a forgotten call site harmless. The call site also checks before building the argument, which avoids diff --git a/examples/capture_content.py b/examples/capture_content.py index dfce35db..b303e0eb 100644 --- a/examples/capture_content.py +++ b/examples/capture_content.py @@ -1,11 +1,13 @@ """ -Example: enable conversation content for every handler at process startup. - -The handlers in the global registry are built with no ``capture_content`` -argument. They read the process default when they run. ``configure`` is -separate from ``init_client``: ``main.py`` initializes the client, and this -example calls ``configure`` once before the first AI invocation. An explicit -``True`` or ``False`` on a factory would override this default. +Example: pin conversation content on for every handler at process startup. + +The built-in default is already on. The handlers in the global registry are +built with no ``capture_content`` argument and read that default when they +run. ``configure`` is separate from ``init_client``: ``main.py`` initializes +the client, and this example calls ``configure`` once before the first AI +invocation so the choice is visible at startup. Pass +``SDKOptions(capture_content=False)`` to opt out. An explicit ``False`` on a +factory removes the transcript everywhere, including offline evaluation spans. Usage (via main.py): python main.py capture-content "" diff --git a/examples/conversation.py b/examples/conversation.py index 73850a6d..efa9143d 100644 --- a/examples/conversation.py +++ b/examples/conversation.py @@ -9,10 +9,10 @@ the same id: root, ``chat``, ``execute_tool``, and the judge's own ``invoke_agent``. 2. Each turn shows a score badge, sourced from the ``gen_ai.evaluation.result`` span event on the judge span. - 3. No judge reasoning anywhere in the telemetry. The score and the judge's config key are - exported; the explanation is not, because it is model prose about the user's conversation - and content attributes require ``capture_content``. The reasoning IS printed below, straight - from ``judge_results`` — that is the caller's copy, and it is unaffected. + 3. The judge explanation is on the span, because capture is on by default. The score and + the judge's config key are exported either way. An explicit ``False`` removes the + explanation from telemetry. The reasoning IS printed below, straight from + ``judge_results`` — that is the caller's copy, and it is unaffected. The flag key must point at an AI Config with a ``judge_configuration``, otherwise there are no judge turns to look at. diff --git a/packages/claude-agents/README.md b/packages/claude-agents/README.md index aaaafa12..3d350692 100644 --- a/packages/claude-agents/README.md +++ b/packages/claude-agents/README.md @@ -40,7 +40,7 @@ asyncio.run(main()) | Option | Default | Description | |---|---|---| -| `capture_content` | `None` | Put prompts, model output, and tool arguments and results on the emitted spans. `None` follows `configure()`, which is off by default. | +| `capture_content` | `None` | Put prompts, model output, and tool arguments and results on the emitted spans. `None` follows `configure()`, which is on by default. An explicit `False` leaves them off. | | `cwd` | the process's working directory | Working directory for the Claude Code process the Agent SDK starts. A `str` or path-like. | | `setting_sources` | `[]` (none) | Which Claude Code settings sources to load: `"user"` (`~/.claude/settings.json`), `"project"` (`.claude/settings.json`, and `CLAUDE.md`), `"local"` (`.claude/settings.local.json`). | diff --git a/packages/claude-agents/agents.md b/packages/claude-agents/agents.md index 22d25000..811fc56b 100644 --- a/packages/claude-agents/agents.md +++ b/packages/claude-agents/agents.md @@ -2,7 +2,7 @@ > **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with > tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler. -> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults). +> Conversation content is on unless the caller passes `capture_content=False`, or the process turns it off. See [Process-wide defaults](../client/README.md#process-wide-defaults). > `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code. diff --git a/packages/claude-agents/src/launchdarkly_ai_claude_agents/handler.py b/packages/claude-agents/src/launchdarkly_ai_claude_agents/handler.py index c2a14d77..0c29c018 100644 --- a/packages/claude-agents/src/launchdarkly_ai_claude_agents/handler.py +++ b/packages/claude-agents/src/launchdarkly_ai_claude_agents/handler.py @@ -711,8 +711,8 @@ def create_claude_agents_handler( Set *capture_content* to put prompts, model output, tool arguments and tool results on the emitted spans. ``None`` (the default) reads the process-wide default at invocation, - which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default - for this handler. See TELEMETRY-CONTRACT.md section 7. + which is on until ``configure`` turns it off. ``True`` or ``False`` overrides that default + for this handler. An explicit ``False`` emits only metadata. See TELEMETRY-CONTRACT.md section 7. *cwd* and *setting_sources* configure the Claude Code process the Agent SDK starts, not the model, so they come from the application and never from a config. Each one given is passed to diff --git a/packages/claude-agents/tests/test_handler.py b/packages/claude-agents/tests/test_handler.py index 0878f3f7..941511a6 100644 --- a/packages/claude-agents/tests/test_handler.py +++ b/packages/claude-agents/tests/test_handler.py @@ -862,7 +862,25 @@ async def test_cache_tokens_folded_into_input( class TestContentCapture: - async def test_emits_no_content_at_all_by_default( + async def test_emits_content_by_default( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr( + handler_mod, + "query", + _fake_query( + [ + assistant_message(content=[TextBlock(text="hi")]), + result_message("hi"), + ] + ), + ) + await create_claude_agents_handler()(BASE_CONFIG, "q") + [chat] = named("chat ") + assert "gen_ai.input.messages" in chat.attributes + assert "gen_ai.output.messages" in chat.attributes + + async def test_emits_no_content_when_capture_content_is_false( self, monkeypatch: pytest.MonkeyPatch ) -> None: monkeypatch.setattr( @@ -872,14 +890,14 @@ async def test_emits_no_content_at_all_by_default( [init_message(tools=["Read"]), assistant_message(), result_message()] ), ) - await create_claude_agents_handler()(TOOL_CONFIG, "q") + await create_claude_agents_handler(capture_content=False)(TOOL_CONFIG, "q") for span in (root(), *named("chat ")): assert "gen_ai.input.messages" not in span.attributes assert "gen_ai.output.messages" not in span.attributes assert "gen_ai.system_instructions" not in span.attributes assert "gen_ai.tool.definitions" not in span.attributes - async def test_root_carries_no_content_by_default_either( + async def test_root_carries_no_output_when_capture_is_false( self, monkeypatch: pytest.MonkeyPatch ) -> None: monkeypatch.setattr( @@ -887,7 +905,7 @@ async def test_root_carries_no_content_by_default_either( "query", _fake_query([assistant_message(), result_message("hi")]), ) - await create_claude_agents_handler()(BASE_CONFIG, "q") + await create_claude_agents_handler(capture_content=False)(BASE_CONFIG, "q") assert "gen_ai.output.messages" not in root().attributes async def test_tool_call_arguments_and_result_gated_by_capture( @@ -925,7 +943,7 @@ async def _query(**kwargs: Any) -> AsyncIterator[Any]: yield result_message("done") monkeypatch.setattr(handler_mod, "query", _query) - await create_claude_agents_handler()( + await create_claude_agents_handler(capture_content=False)( TOOL_CONFIG, "q", {"search": lambda _: "r"} ) [tool_span] = named("execute_tool ") @@ -1610,7 +1628,11 @@ async def test_ld_span_attributes_land_on_root_only( assert attrs["launchdarkly.config.key"] == "cfg" assert attrs["launchdarkly.variation.key"] == "var" assert attrs["launchdarkly.run.id"] == "run-1" - assert [e.name for e in root().events] == ["feature_flag"] + assert [ + event.name + for event in root().events + if not event.name.startswith("gen_ai.content") + ] == ["feature_flag"] class TestModelParametersForwarding: diff --git a/packages/claude-messages/agents.md b/packages/claude-messages/agents.md index 6ca5d3c5..5ab1b06e 100644 --- a/packages/claude-messages/agents.md +++ b/packages/claude-messages/agents.md @@ -2,7 +2,7 @@ > **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with > tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler. -> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults). +> Conversation content is on unless the caller passes `capture_content=False`, or the process turns it off. See [Process-wide defaults](../client/README.md#process-wide-defaults). > `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code. diff --git a/packages/claude-messages/src/launchdarkly_ai_claude_messages/handler.py b/packages/claude-messages/src/launchdarkly_ai_claude_messages/handler.py index ebd17db4..9bef7de7 100644 --- a/packages/claude-messages/src/launchdarkly_ai_claude_messages/handler.py +++ b/packages/claude-messages/src/launchdarkly_ai_claude_messages/handler.py @@ -537,9 +537,9 @@ def create_claude_messages_handler( Set *capture_content* to put prompts, model output, tool arguments and tool results on the emitted spans. ``None`` (the default) reads the process-wide default at invocation, - which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default - for this handler. Conversation content is PII, so a run emits only metadata, meaning - models, token counts, timings and tool names, until capture is turned on. Turning this + which is on until ``configure`` turns it off. ``True`` or ``False`` overrides that default + for this handler. Conversation content is PII. An explicit ``False`` emits only metadata, + meaning models, token counts, timings and tool names. Turning this on sends the text of every request and response to whatever collector the SDK is pointed at. """ report_usage( diff --git a/packages/claude-messages/tests/test_handler.py b/packages/claude-messages/tests/test_handler.py index 3732a860..9f246aa6 100644 --- a/packages/claude-messages/tests/test_handler.py +++ b/packages/claude-messages/tests/test_handler.py @@ -1007,7 +1007,9 @@ async def test_carries_the_launchdarkly_attributes_and_feature_flag_event( assert attrs["launchdarkly.variation.key"] == "v" assert attrs["launchdarkly.run.id"] == "r" assert attrs["launchdarkly.graph.key"] == "g" - assert [n for n, _ in rec.root.events] == ["feature_flag"] + assert [ + name for name, _ in rec.root.events if not name.startswith("gen_ai.content") + ] == ["feature_flag"] async def test_child_spans_carry_no_launchdarkly_identity( self, mock_anthropic: MagicMock @@ -1188,15 +1190,29 @@ async def test_sets_status_ok_on_a_successful_turn( class TestContentCapture: """TELEMETRY-CONTRACT.md section 7.""" - async def test_emits_no_content_at_all_by_default( + async def test_emits_content_by_default(self, mock_anthropic: MagicMock) -> None: + ctx, rec = _recording() + from launchdarkly_ai_claude_messages import create_claude_messages_handler + + with ctx: + await create_claude_messages_handler()(CONFIG, "q", {}, {}) + assert any( + "gen_ai.input.messages" in span.attributes + or any(key.startswith("gen_ai.prompt") for key in span.attributes) + for span in rec.spans + ) + + async def test_emits_no_content_when_capture_content_is_false( self, mock_anthropic: MagicMock ) -> None: - # Conversation content is PII. This is the assertion worth pinning hardest. + # An explicit false still strips conversation content. This is the opt-out. ctx, rec = _recording() from launchdarkly_ai_claude_messages import create_claude_messages_handler with ctx: - await create_claude_messages_handler()(CONFIG, "q", {}, {}) + await create_claude_messages_handler(capture_content=False)( + CONFIG, "q", {}, {} + ) for span in rec.spans: content_keys = [ k @@ -2089,7 +2105,7 @@ async def test_fails_the_spans_when_the_stream_raises( assert StatusCode.ERROR in rec.root.statuses assert rec.root.ended == 1 - async def test_emits_no_content_by_default_on_the_streaming_path( + async def test_emits_content_by_default_on_the_streaming_path( self, mock_anthropic: MagicMock ) -> None: self._patch_stream(mock_anthropic, ["hi"]) @@ -2099,6 +2115,20 @@ async def test_emits_no_content_by_default_on_the_streaming_path( with ctx: async for _ in await create_claude_messages_handler().stream(CONFIG, "q"): pass + assert any("gen_ai.input.messages" in span.attributes for span in rec.spans) + + async def test_emits_no_content_when_capture_is_false_on_the_streaming_path( + self, mock_anthropic: MagicMock + ) -> None: + self._patch_stream(mock_anthropic, ["hi"]) + ctx, rec = _recording() + from launchdarkly_ai_claude_messages import create_claude_messages_handler + + with ctx: + async for _ in await create_claude_messages_handler( + capture_content=False + ).stream(CONFIG, "q"): + pass for span in rec.spans: assert [k for k in span.attributes if k.startswith("gen_ai.prompt")] == [] assert [n for n, _ in span.events if n.startswith("gen_ai.content")] == [] diff --git a/packages/client/README.md b/packages/client/README.md index 81d1bf61..6ca08eff 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -52,7 +52,7 @@ No code changes are required — `init_client()` detects the packages at runtime The evaluations harness reads an LD-hosted dataset, creates a new evaluation and API-source run, and invokes your handler once per row. Rows can then be scored by LaunchDarkly judges and local scorer functions; see [Score rows with judges and scorers](#score-rows-with-judges-and-scorers). Each success or error queues a `$ld:ai:offline-evals:generation` custom event containing the evaluation, run, dataset, and row identifiers plus output or error (`errorMessage` is included for `ERROR` rows), nested `usage.inputTokens`/`usage.outputTokens`, timing, and stable hashes. Dataset-owned input, expected output, metadata, and variables are not duplicated in the event. Each queued event is logged at `INFO` on the `launchdarkly_ai_server.evaluations.runner` logger with its RFC3339 UTC `emittedAt` timestamp and stable `eventId`, making it possible to compare SDK emission time with ClickHouse arrival time once that logger is enabled. The same `emittedAt` value is included in the event payload. Events are flushed before the summary is fetched and the call returns; handlers are never rerun to retry event delivery. Pass/fail is derived from LaunchDarkly's run summary. -Each row is also exported as one trace on the tracer provider `init_client` registered: a `launchdarkly.dataset_row` root with the generation, its tool calls, and each criterion's `evaluate` span beneath it, all sharing the row's `gen_ai.conversation.id`. A provider handler traces its own call; for a plain function the runner writes the `chat` and `execute_tool` spans. Prompts, output, tool payloads, and judge explanations follow the same [`capture_content`](#process-wide-defaults) gate as every other span, so they are omitted unless capture is on. +Each row is also exported as one trace on the tracer provider `init_client` registered: a `launchdarkly.dataset_row` root with the generation, its tool calls, and each criterion's `evaluate` span beneath it, all sharing the row's `gen_ai.conversation.id`. A provider handler traces its own call; for a plain function the runner writes the `chat` and `execute_tool` spans. Prompts, output, tool payloads, and judge explanations follow the same [`capture_content`](#process-wide-defaults) gate as every other span. The built-in default sends them. Setting capture to false removes them, including on these evaluation spans. The `$ld:ai:offline-evals:*` events still carry no transcript. Management API requests go to the `api_base_uri` option, then `LD_API_BASE_URI`, then `https://app.launchdarkly.com`; `LD_BASE_URI` is never used, because it may point at a relay proxy. Result links use `ui_base_uri`, then `LD_UI_BASE_URI`, then `https://app.launchdarkly.com`; this is independent of `LD_API_BASE_URI`. At either level a blank value counts as unset, and a trailing `/` is stripped. After flushing generation events, the harness polls the run summary endpoint until passed + failed + error rows fully account for a nonzero total with no pending rows, polling every `poll_interval_seconds` (default 2s) up to `poll_timeout_seconds` (default 180s); pass either to `run()` to widen both for large datasets. The summary endpoint does not return run state, so `RunSummary` exposes row counts only. A run passes only when the completed summary has no failed, error, or pending rows. `failed_rows` counts rows whose criteria were scored and did not meet their threshold, so a gate that ignored it would exit 0 on a run where every row failed its judge. Evaluation keys must be unique because every call creates a new evaluation with `POST`. @@ -250,13 +250,14 @@ warning and ignored. ### Process-wide defaults -`configure` sets behavior for the process. It does not initialize the LaunchDarkly client; call `init_client` for that. Handlers built without an explicit `capture_content` read this value when they run, so one startup call covers every handler. An explicit `True` or `False` on a handler factory or convenience wrapper overrides it. Call `configure` from process startup, not from request data. It can be replaced until the first AI invocation; after that only the same options are accepted. `shutdown()` restores the built-in default, which is off. +`configure` sets behavior for the process. It does not initialize the LaunchDarkly client; call `init_client` for that. Handlers built without an explicit `capture_content` read this value when they run, so one startup call covers every handler. The built-in default is on: prompts, model output, tool payloads, and judge explanations are sent unless you turn capture off. An explicit `True` or `False` on a handler factory or convenience wrapper overrides it, and an explicit `False` removes that text everywhere, including on offline evaluation spans. Call `configure` from process startup, not from request data. It can be replaced until the first AI invocation; after that only the same options are accepted. `shutdown()` restores the built-in default, which is on. ```python from launchdarkly_ai_server import SDKOptions, configure, init_client, shutdown async def main(): - configure(options=SDKOptions(capture_content=True)) + # Capture is on unless you turn it off. + configure(options=SDKOptions(capture_content=False)) await init_client({"sdkKey": "sdk-...", "serviceName": "my-service"}) # create_*_handler() with no capture_content argument follows the default above. await shutdown() diff --git a/packages/client/agents.md b/packages/client/agents.md index b93877a7..a4d995ed 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -875,7 +875,7 @@ When `enabled` is `False`, `config` is always `None`. When `enabled` is `True` b ## Lifecycle Invariants -- **Process defaults.** `configure(options=SDKOptions(capture_content=True))` sets behavior defaults for the process. Call it once at startup, not from request data, and not via `init_client`. Handlers omit `capture_content` to follow it. `shutdown()` restores the built-in defaults. +- **Process defaults.** The built-in `capture_content` default is on. `configure(options=SDKOptions(capture_content=False))` turns it off for the process. Call `configure` once at startup, not from request data, and not via `init_client`. Handlers omit `capture_content` to follow it. An explicit `False` strips content everywhere, including offline evaluation spans. `shutdown()` restores the built-in default, which is on. - **Lazy initialization.** Importing the package does not initialize the LD client. The first API call that needs LaunchDarkly calls `init_client()` internally when `LD_SDK_KEY` is set. - **Explicit initialization — SDK path.** `await init_client(options?)` dynamically imports `launchdarkly-server-sdk` at runtime (optional peer dep). If the package is not installed it raises with a clear message. - **Explicit initialization — BYOC path.** `await init_client(client)` accepts any pre-initialized object that satisfies `LDClientInterface` — this is the path for custom or edge environments whose SDK has different init semantics. diff --git a/packages/client/api-report/launchdarkly_ai_server.txt b/packages/client/api-report/launchdarkly_ai_server.txt index d1792118..e4e6263d 100644 --- a/packages/client/api-report/launchdarkly_ai_server.txt +++ b/packages/client/api-report/launchdarkly_ai_server.txt @@ -184,9 +184,9 @@ RunUsage [class, @dataclass] SDK_INFO_CONTEXT: LDContext = {'kind': 'ld_ai', 'key': 'ld-internal-tracking', 'anonymous': True} SDK_INFO_EVENT = '$ld:ai:sdk:info' SDKOptions [class, @dataclass(frozen=True)] - __init__(capture_content: bool = False) + __init__(capture_content: bool = True) __post_init__() -> None - capture_content: bool = False + capture_content: bool = True set_conversation_id_if_absent(span: Any, conversation: str) -> None set_input_content_attributes(span: Any, capture: bool, *, system_instructions: str | None = None, messages: list[SpanMessage] | None = None, tool_definitions: list[ToolDefinitionInput] | None = None) -> None set_ld_span_attributes(span: Any, variables: dict[str, Any] | None) -> None diff --git a/packages/client/src/launchdarkly_ai_server/evaluations/spans.py b/packages/client/src/launchdarkly_ai_server/evaluations/spans.py index 14fbf559..ecdd7110 100644 --- a/packages/client/src/launchdarkly_ai_server/evaluations/spans.py +++ b/packages/client/src/launchdarkly_ai_server/evaluations/spans.py @@ -13,8 +13,8 @@ These spans export to the customer's project on the SDK key. They follow the same content gate as every other span: the handler's explicit override when it -has one, otherwise the process default, which is off. An explicit ``True`` -exports the transcript on these spans too. The ``$ld:ai:offline-evals:*`` +has one, otherwise the process default, which is on. An explicit ``False`` +strips the transcript on these spans too. The ``$ld:ai:offline-evals:*`` events never carry the transcript. """ diff --git a/packages/client/src/launchdarkly_ai_server/lifecycle.py b/packages/client/src/launchdarkly_ai_server/lifecycle.py index f3c44a9d..5842d318 100644 --- a/packages/client/src/launchdarkly_ai_server/lifecycle.py +++ b/packages/client/src/launchdarkly_ai_server/lifecycle.py @@ -83,7 +83,8 @@ def effective_capture_content(override: bool | None) -> bool: An explicit ``True`` or ``False`` — from a convenience wrapper, then from the handler factory — wins. ``None`` reads the configured default, which - itself defaults to ``False``. + itself defaults to ``True``. An explicit ``False`` removes conversation + content everywhere, including on offline evaluation spans. """ if override is not None: return override diff --git a/packages/client/src/launchdarkly_ai_server/types.py b/packages/client/src/launchdarkly_ai_server/types.py index 45139325..522bd71f 100644 --- a/packages/client/src/launchdarkly_ai_server/types.py +++ b/packages/client/src/launchdarkly_ai_server/types.py @@ -627,13 +627,15 @@ class SDKOptions: """Process-wide behavior defaults, applied by :func:`configure`. ``capture_content`` is the default for writing prompts, model output, tool - arguments, tool results, and judge explanations. It is a default, not a - ceiling: an explicit ``True`` or ``False`` on a handler factory or a - convenience wrapper overrides it. Further process-wide defaults are added - as fields on this object. + arguments, tool results, and judge explanations. The built-in value is + ``True``: a process that never calls ``configure`` sends that text. It is a + default, not a ceiling: an explicit ``True`` or ``False`` on a handler + factory or a convenience wrapper overrides it, and an explicit ``False`` + removes the text everywhere, including on offline evaluation spans. Further + process-wide defaults are added as fields on this object. """ - capture_content: bool = False + capture_content: bool = True def __post_init__(self) -> None: if not isinstance(self.capture_content, bool): diff --git a/packages/client/tests/test_configure.py b/packages/client/tests/test_configure.py index 53f0fe5b..4f0ab359 100644 --- a/packages/client/tests/test_configure.py +++ b/packages/client/tests/test_configure.py @@ -45,14 +45,15 @@ def _handler(capture_content: bool | None = None) -> ProviderHandler: return ProviderHandler(_fn, capture_content=capture_content) -def test_builtin_default_is_off() -> None: - assert _handler().capture_content is False +def test_builtin_default_is_on() -> None: + assert SDKOptions().capture_content is True + assert _handler().capture_content is True def test_configure_is_read_after_the_handler_is_built() -> None: handler = _handler() - configure(options=SDKOptions(capture_content=True)) - assert handler.capture_content is True + configure(options=SDKOptions(capture_content=False)) + assert handler.capture_content is False def test_configure_can_be_replaced_before_an_invocation() -> None: @@ -101,7 +102,7 @@ def test_sdk_options_is_frozen() -> None: async def test_a_changed_configuration_is_rejected_after_invocation() -> None: await _handler()({}) with pytest.raises(RuntimeError, match="cannot change"): - configure(options=SDKOptions(capture_content=True)) + configure(options=SDKOptions(capture_content=False)) async def test_an_identical_configuration_is_allowed_after_invocation() -> None: @@ -119,26 +120,26 @@ async def _stream(*_args: Any, **_kwargs: Any) -> Any: gen = await handler.stream({}) await gen.aclose() with pytest.raises(RuntimeError, match="cannot change"): - configure(options=SDKOptions(capture_content=True)) + configure(options=SDKOptions(capture_content=False)) async def test_a_missing_stream_does_not_latch() -> None: with pytest.raises(NotImplementedError): await _handler().stream({}) - configure(options=SDKOptions(capture_content=True)) - assert _handler().capture_content is True + configure(options=SDKOptions(capture_content=False)) + assert _handler().capture_content is False async def test_shutdown_restores_the_builtin_default() -> None: - configure(options=SDKOptions(capture_content=True)) + configure(options=SDKOptions(capture_content=False)) await _handler()({}) await shutdown() - assert _handler().capture_content is False - configure(options=SDKOptions(capture_content=True)) assert _handler().capture_content is True + configure(options=SDKOptions(capture_content=False)) + assert _handler().capture_content is False -async def test_run_judges_writes_the_explanation_from_the_configured_default() -> None: +async def test_run_judges_writes_the_explanation_from_the_builtin_default() -> None: """The judge explanation follows the handler's effective capture flag. Goes through ``run_judges``, not a copy of the ``if capture_content`` @@ -191,7 +192,6 @@ async def fn( } ) lifecycle_module._set_client_for_testing(client) - configure(options=SDKOptions(capture_content=True)) parent = { "model": {"name": "gpt-4"}, "provider": {"name": "TestProvider"}, @@ -243,6 +243,6 @@ async def test_init_client_does_not_carry_behavior_defaults( patch.object(lifecycle_module, "_setup_telemetry", return_value=None), caplog.at_level("WARNING", logger="launchdarkly_ai_server.lifecycle"), ): - await init_client({"capture_content": True}, stub) - assert _handler().capture_content is False + await init_client({"capture_content": False}, stub) + assert _handler().capture_content is True assert any("capture_content" in record.getMessage() for record in caplog.records) diff --git a/packages/client/tests/test_evaluations_spans.py b/packages/client/tests/test_evaluations_spans.py index 4dd806d3..914d257f 100644 --- a/packages/client/tests/test_evaluations_spans.py +++ b/packages/client/tests/test_evaluations_spans.py @@ -268,7 +268,7 @@ def lookup_order_tool() -> EvalTool: @pytest.fixture def capture_on() -> Iterator[None]: - """Transcript-shape cases run with capture on. The built-in default is off.""" + """Transcript-shape cases pin capture on, independent of the process default.""" _reset_for_testing() configure(options=SDKOptions(capture_content=True)) yield @@ -831,41 +831,17 @@ def _write_captured_row(*, capture: bool | None = None) -> None: def test_offline_spans_follow_the_capture_content_gate( span_export: tuple[InMemorySpanExporter, list[str]], ) -> None: - """Evaluation spans omit the transcript by default and export it when capture is on. + """Evaluation spans export the transcript by default and omit it when capture is off. - An explicit true on the call still writes it when the process default is false. + An explicit false on the call removes it while the process default is on. An explicit + true on the call still writes it when the process default is false. """ exporter, _flushed = span_export _reset_for_testing() try: - assert SDKOptions().capture_content is False - assert effective_capture_content(None) is False - - _write_captured_row() - by_name = spans_by_name(exporter) - for attributes in ( - by_name["chat gpt-4o"][0].attributes or {}, - by_name["execute_tool lookup"][0].attributes or {}, - ): - for key in _CONTENT_KEYS: - assert key not in attributes - result = next( - event.attributes - for event in by_name["evaluate accuracy"][0].events - if event.name == "gen_ai.evaluation.result" - ) - assert result is not None - assert "gen_ai.evaluation.explanation" not in result - assert result["gen_ai.evaluation.name"] == "accuracy" - assert result["gen_ai.evaluation.score.value"] == 0.9 - assert not any( - "Paris" in str(value) - for value in (by_name["chat gpt-4o"][0].attributes or {}).values() - ) - - exporter.clear() - configure(options=SDKOptions(capture_content=True)) + assert SDKOptions().capture_content is True assert effective_capture_content(None) is True + _write_captured_row() by_name = spans_by_name(exporter) chat_attributes = by_name["chat gpt-4o"][0].attributes or {} @@ -887,9 +863,54 @@ def test_offline_spans_follow_the_capture_content_gate( ) assert explanation is not None assert explanation["gen_ai.evaluation.explanation"] == "matches the capital" + assert explanation["gen_ai.evaluation.name"] == "accuracy" + assert explanation["gen_ai.evaluation.score.value"] == 0.9 + + exporter.clear() + _write_captured_row(capture=False) + by_name = spans_by_name(exporter) + for attributes in ( + by_name["chat gpt-4o"][0].attributes or {}, + by_name["execute_tool lookup"][0].attributes or {}, + ): + for key in _CONTENT_KEYS: + assert key not in attributes + stripped = next( + event.attributes + for event in by_name["evaluate accuracy"][0].events + if event.name == "gen_ai.evaluation.result" + ) + assert stripped is not None + assert "gen_ai.evaluation.explanation" not in stripped + assert stripped["gen_ai.evaluation.name"] == "accuracy" + assert stripped["gen_ai.evaluation.score.value"] == 0.9 exporter.clear() configure(options=SDKOptions(capture_content=False)) + assert effective_capture_content(None) is False + _write_captured_row() + by_name = spans_by_name(exporter) + for attributes in ( + by_name["chat gpt-4o"][0].attributes or {}, + by_name["execute_tool lookup"][0].attributes or {}, + ): + for key in _CONTENT_KEYS: + assert key not in attributes + result = next( + event.attributes + for event in by_name["evaluate accuracy"][0].events + if event.name == "gen_ai.evaluation.result" + ) + assert result is not None + assert "gen_ai.evaluation.explanation" not in result + assert result["gen_ai.evaluation.name"] == "accuracy" + assert result["gen_ai.evaluation.score.value"] == 0.9 + assert not any( + "Paris" in str(value) + for value in (by_name["chat gpt-4o"][0].attributes or {}).values() + ) + + exporter.clear() _write_captured_row(capture=True) kept = spans_by_name(exporter)["chat gpt-4o"][0].attributes or {} assert "Where is Paris?" in str(kept["gen_ai.input.messages"]) diff --git a/packages/langchain-agents/agents.md b/packages/langchain-agents/agents.md index 2e4187ec..7d769485 100644 --- a/packages/langchain-agents/agents.md +++ b/packages/langchain-agents/agents.md @@ -2,7 +2,7 @@ > **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with > tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler. -> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults). +> Conversation content is on unless the caller passes `capture_content=False`, or the process turns it off. See [Process-wide defaults](../client/README.md#process-wide-defaults). > `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code. diff --git a/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py b/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py index 5346a412..3ba0c188 100644 --- a/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py +++ b/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py @@ -641,9 +641,9 @@ def create_langchain_agents_handler( Set *capture_content* to put prompts, model output, tool arguments and tool results on the emitted spans. ``None`` (the default) reads the process-wide default at invocation, - which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default - for this handler. Conversation content is PII, so a run emits only metadata, meaning - models, token counts, timings and tool names, until capture is turned on. + which is on until ``configure`` turns it off. ``True`` or ``False`` overrides that default + for this handler. Conversation content is PII. An explicit ``False`` emits only metadata, + meaning models, token counts, timings and tool names. """ report_usage( "langchain-agents.createLangChainAgentsHandler", PACKAGE_NAME, __version__ diff --git a/packages/langchain-agents/tests/test_handler.py b/packages/langchain-agents/tests/test_handler.py index a68c42a4..e48dea70 100644 --- a/packages/langchain-agents/tests/test_handler.py +++ b/packages/langchain-agents/tests/test_handler.py @@ -996,7 +996,9 @@ async def test_carries_the_launchdarkly_attributes_and_feature_flag_event( assert attrs["launchdarkly.variation.key"] == "v" assert attrs["launchdarkly.run.id"] == "r" assert attrs["launchdarkly.graph.key"] == "g" - assert [n for n, _ in rec.root.events] == ["feature_flag"] + assert [ + name for name, _ in rec.root.events if not name.startswith("gen_ai.content") + ] == ["feature_flag"] @pytest.mark.asyncio async def test_child_spans_carry_no_launchdarkly_identity(self) -> None: @@ -1136,11 +1138,24 @@ class TestContentCapture: """TELEMETRY-CONTRACT.md section 7.""" @pytest.mark.asyncio - async def test_emits_no_content_at_all_by_default(self) -> None: + async def test_emits_content_by_default(self) -> None: ctx, rec = _recording() llm = _FakeToolModel(replies=[_ai_message("hi")]) with ctx: await create_langchain_agents_handler(llm)(BASE_CONFIG, "q") + assert any( + "gen_ai.input.messages" in span.attributes + or any(key.startswith("gen_ai.prompt") for key in span.attributes) + for span in rec.spans + ) + + async def test_emits_no_content_when_capture_content_is_false(self) -> None: + ctx, rec = _recording() + llm = _FakeToolModel(replies=[_ai_message("hi")]) + with ctx: + await create_langchain_agents_handler(llm, capture_content=False)( + BASE_CONFIG, "q" + ) for span in rec.spans: content_keys = [ k @@ -1682,7 +1697,7 @@ async def test_fails_the_spans_when_the_stream_raises(self) -> None: assert rec.root.ended == 1 @pytest.mark.asyncio - async def test_emits_no_content_by_default_on_the_streaming_path(self) -> None: + async def test_emits_content_by_default_on_the_streaming_path(self) -> None: ctx, rec = _recording() llm = _FakeToolModel(replies=[_ai_message("hi")]) with ctx: @@ -1690,6 +1705,22 @@ async def test_emits_no_content_by_default_on_the_streaming_path(self) -> None: BASE_CONFIG, "q", {}, {} ): pass + assert any( + "gen_ai.input.messages" in span.attributes + or any(key.startswith("gen_ai.prompt") for key in span.attributes) + for span in rec.spans + ) + + async def test_emits_no_content_when_capture_is_false_on_the_streaming_path( + self, + ) -> None: + ctx, rec = _recording() + llm = _FakeToolModel(replies=[_ai_message("hi")]) + with ctx: + async for _ in await create_langchain_agents_handler( + llm, capture_content=False + ).stream(BASE_CONFIG, "q", {}, {}): + pass for span in rec.spans: assert [k for k in span.attributes if k.startswith("gen_ai.prompt")] == [] assert [n for n, _ in span.events if n.startswith("gen_ai.content")] == [] diff --git a/packages/langchain-messages/agents.md b/packages/langchain-messages/agents.md index 6c889d95..a54e7ade 100644 --- a/packages/langchain-messages/agents.md +++ b/packages/langchain-messages/agents.md @@ -2,7 +2,7 @@ > **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with > tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler. -> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults). +> Conversation content is on unless the caller passes `capture_content=False`, or the process turns it off. See [Process-wide defaults](../client/README.md#process-wide-defaults). > `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code. diff --git a/packages/langchain-messages/src/launchdarkly_ai_langchain_messages/handler.py b/packages/langchain-messages/src/launchdarkly_ai_langchain_messages/handler.py index 78f36622..48e11285 100644 --- a/packages/langchain-messages/src/launchdarkly_ai_langchain_messages/handler.py +++ b/packages/langchain-messages/src/launchdarkly_ai_langchain_messages/handler.py @@ -774,9 +774,9 @@ def create_langchain_messages_handler( Set *capture_content* to put prompts, model output, tool arguments and tool results on the emitted spans. ``None`` (the default) reads the process-wide default at invocation, - which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default - for this handler. Conversation content is PII, so a run emits only metadata, meaning - models, token counts, timings and tool names, until capture is turned on. + which is on until ``configure`` turns it off. ``True`` or ``False`` overrides that default + for this handler. Conversation content is PII. An explicit ``False`` emits only metadata, + meaning models, token counts, timings and tool names. """ report_usage("langchain-messages.createLangChainHandler", PACKAGE_NAME, __version__) return _create_langchain_messages_handler(llm, capture_content=capture_content) diff --git a/packages/langchain-messages/tests/test_handler.py b/packages/langchain-messages/tests/test_handler.py index d18aed5f..9ea88ea3 100644 --- a/packages/langchain-messages/tests/test_handler.py +++ b/packages/langchain-messages/tests/test_handler.py @@ -674,7 +674,9 @@ async def test_carries_the_launchdarkly_attributes_and_feature_flag_event( assert attrs["launchdarkly.variation.key"] == "v" assert attrs["launchdarkly.run.id"] == "r" assert attrs["launchdarkly.graph.key"] == "g" - assert [n for n, _ in rec.root.events] == ["feature_flag"] + assert [ + name for name, _ in rec.root.events if not name.startswith("gen_ai.content") + ] == ["feature_flag"] async def test_child_spans_carry_no_launchdarkly_identity(self) -> None: ctx, rec = _recording() @@ -824,7 +826,7 @@ async def test_sets_status_ok_on_a_successful_turn(self) -> None: class TestContentCapture: - async def test_emits_no_content_at_all_by_default(self) -> None: + async def test_emits_content_by_default(self) -> None: ctx, rec = _recording() from launchdarkly_ai_langchain_messages import create_langchain_messages_handler @@ -832,6 +834,20 @@ async def test_emits_no_content_at_all_by_default(self) -> None: await create_langchain_messages_handler(llm=_make_llm())( CONFIG, "q", {}, {} ) + assert any( + "gen_ai.input.messages" in span.attributes + or any(key.startswith("gen_ai.prompt") for key in span.attributes) + for span in rec.spans + ) + + async def test_emits_no_content_when_capture_content_is_false(self) -> None: + ctx, rec = _recording() + from launchdarkly_ai_langchain_messages import create_langchain_messages_handler + + with ctx: + await create_langchain_messages_handler( + llm=_make_llm(), capture_content=False + )(CONFIG, "q", {}, {}) for span in rec.spans: content_keys = [ k @@ -1601,7 +1617,7 @@ async def _astream(msgs: Any) -> AsyncGenerator[Any, None]: assert StatusCode.ERROR in rec.root.statuses assert rec.root.ended == 1 - async def test_emits_no_content_by_default_on_the_streaming_path(self) -> None: + async def test_emits_content_by_default_on_the_streaming_path(self) -> None: ctx, rec = _recording() from launchdarkly_ai_langchain_messages import create_langchain_messages_handler @@ -1611,6 +1627,24 @@ async def test_emits_no_content_by_default_on_the_streaming_path(self) -> None: CONFIG, "q" ): pass + assert any( + "gen_ai.input.messages" in span.attributes + or any(key.startswith("gen_ai.prompt") for key in span.attributes) + for span in rec.spans + ) + + async def test_emits_no_content_when_capture_is_false_on_the_streaming_path( + self, + ) -> None: + ctx, rec = _recording() + from launchdarkly_ai_langchain_messages import create_langchain_messages_handler + + llm = self._make_streaming_llm(["hi"]) + with ctx: + async for _ in await create_langchain_messages_handler( + llm=llm, capture_content=False + ).stream(CONFIG, "q"): + pass for span in rec.spans: assert [k for k in span.attributes if k.startswith("gen_ai.prompt")] == [] assert [n for n, _ in span.events if n.startswith("gen_ai.content")] == [] @@ -2200,9 +2234,9 @@ class _Unserialisable: ctx, rec = _recording() with ctx: - result = await create_langchain_messages_handler(llm=llm)( - config, "q", {"myTool": AsyncMock(return_value="r")}, {} - ) + result = await create_langchain_messages_handler( + llm=llm, capture_content=False + )(config, "q", {"myTool": AsyncMock(return_value="r")}, {}) assert result["output"] is parsed assert "gen_ai.output.messages" not in rec.root.attributes @@ -2229,9 +2263,9 @@ class _Unserialisable: ctx, rec = _recording() with ctx: - result = await create_langchain_messages_handler(llm=llm)( - config, "q", {}, {} - ) + result = await create_langchain_messages_handler( + llm=llm, capture_content=False + )(config, "q", {}, {}) assert result["output"] is parsed assert result["usage"] == {"input_tokens": 8, "output_tokens": 2} assert "gen_ai.output.messages" not in rec.root.attributes @@ -2556,7 +2590,9 @@ def _counting(span: Any, capture: bool, *a: Any, **k: Any) -> None: monkeypatch.setattr(handler_mod, "set_output_content_attributes", _counting) - h = create_langchain_messages_handler(llm=self._streaming_llm(["hi"])) + h = create_langchain_messages_handler( + llm=self._streaming_llm(["hi"]), capture_content=False + ) async for _ in await h.stream(CONFIG, "q"): pass diff --git a/packages/litellm-agents/tests/test_handler.py b/packages/litellm-agents/tests/test_handler.py index 36a5cea0..09c0838e 100644 --- a/packages/litellm-agents/tests/test_handler.py +++ b/packages/litellm-agents/tests/test_handler.py @@ -148,7 +148,7 @@ def test_wildcard_metadata_and_capture_content(self) -> None: handler = create_litellm_agents_handler(capture_content=True) assert handler.provides_for == ("*", "agent") assert handler.capture_content is True - assert create_litellm_agents_handler().capture_content is False + assert create_litellm_agents_handler().capture_content is True async def test_evaluated_model_constructs_agents_sdk_litellm_model(self) -> None: agents = _agents_module() @@ -493,9 +493,9 @@ async def test_root_identity_and_content_capture_gating(self) -> None: "name": "Ada", } with _patch_agents(agents), patch.object(handler_mod, "trace", trace): - await create_litellm_agents_handler(model_factory=lambda *_: object())( - CONFIG, "private", {}, variables - ) + await create_litellm_agents_handler( + model_factory=lambda *_: object(), capture_content=False + )(CONFIG, "private", {}, variables) attributes = { call.args[0]: call.args[1] for call in span.set_attribute.call_args_list } diff --git a/packages/litellm-messages/tests/test_handler.py b/packages/litellm-messages/tests/test_handler.py index 5f99d772..839d6d66 100644 --- a/packages/litellm-messages/tests/test_handler.py +++ b/packages/litellm-messages/tests/test_handler.py @@ -154,7 +154,7 @@ def test_wildcard_metadata_and_capture_content(self) -> None: handler = create_litellm_messages_handler(capture_content=True) assert handler.provides_for == ("*", "messages") assert handler.capture_content is True - assert create_litellm_messages_handler().capture_content is False + assert create_litellm_messages_handler().capture_content is True async def test_calls_in_process_litellm_acompletion(self) -> None: completion = AsyncMock(return_value=_response()) @@ -604,9 +604,9 @@ async def test_identity_and_content_are_gated(self) -> None: "ldContext": {"kind": "user", "key": "ada"}, } with patch.object(handler_mod, "trace", trace): - await create_litellm_messages_handler(completion=completion)( - CONFIG, "private prompt", {}, variables - ) + await create_litellm_messages_handler( + completion=completion, capture_content=False + )(CONFIG, "private prompt", {}, variables) attributes = { call.args[0]: call.args[1] for call in span.set_attribute.call_args_list } diff --git a/packages/openai-agents/agents.md b/packages/openai-agents/agents.md index 3393c4a4..602c25ce 100644 --- a/packages/openai-agents/agents.md +++ b/packages/openai-agents/agents.md @@ -2,7 +2,7 @@ > **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with > tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler. -> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults). +> Conversation content is on unless the caller passes `capture_content=False`, or the process turns it off. See [Process-wide defaults](../client/README.md#process-wide-defaults). > `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code. diff --git a/packages/openai-agents/src/launchdarkly_ai_openai_agents/handler.py b/packages/openai-agents/src/launchdarkly_ai_openai_agents/handler.py index b0fe0942..bcaf0f02 100644 --- a/packages/openai-agents/src/launchdarkly_ai_openai_agents/handler.py +++ b/packages/openai-agents/src/launchdarkly_ai_openai_agents/handler.py @@ -653,9 +653,9 @@ def create_openai_agent_handler( Set *capture_content* to put prompts, model output, tool arguments and tool results on the emitted spans. ``None`` (the default) reads the process-wide default at invocation, - which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default - for this handler. Conversation content is PII, so a run emits only metadata, meaning - models, token counts, timings and tool names, until capture is turned on. + which is on until ``configure`` turns it off. ``True`` or ``False`` overrides that default + for this handler. Conversation content is PII. An explicit ``False`` emits only metadata, + meaning models, token counts, timings and tool names. """ report_usage("openai-agents.createOpenAIAgentHandler", PACKAGE_NAME, __version__) return _create_openai_agent_handler(capture_content=capture_content) diff --git a/packages/openai-agents/tests/test_handler.py b/packages/openai-agents/tests/test_handler.py index 9a0d6ced..ab85a7bf 100644 --- a/packages/openai-agents/tests/test_handler.py +++ b/packages/openai-agents/tests/test_handler.py @@ -774,7 +774,9 @@ async def test_children_carry_no_launchdarkly_attributes(self) -> None: ) for span in rec.spans[1:]: assert [k for k in span.attributes if k.startswith("launchdarkly.")] == [] - assert [n for n, _ in rec.root.events] == ["feature_flag"] + assert [ + name for name, _ in rec.root.events if not name.startswith("gen_ai.content") + ] == ["feature_flag"] assert all( "feature_flag" not in [n for n, _ in s.events] for s in rec.spans[1:] ) @@ -838,7 +840,7 @@ async def test_feature_flag_event_on_root_only(self) -> None: } with ctx, _patched_agents(agents_mod): await create_openai_agent_handler()(CONFIG, "q", {}, vs) - assert rec.root.events == [ + assert [event for event in rec.root.events if event[0] == "feature_flag"] == [ ( "feature_flag", { @@ -991,13 +993,28 @@ async def test_fails_an_open_tool_span_when_the_run_crashes_mid_flight( class TestContentCapture: - async def test_emits_no_content_at_all_by_default(self) -> None: + async def test_emits_content_by_default(self) -> None: ctx, rec = _recording() agents_mod = _fake_agents_module( run=_make_run([{"output": _text_output("answer")}]) ) with ctx, _patched_agents(agents_mod): await create_openai_agent_handler()(CONFIG, "my question", {}, {}) + assert any( + "gen_ai.input.messages" in span.attributes + or any(key.startswith("gen_ai.prompt") for key in span.attributes) + for span in rec.spans + ) + + async def test_emits_no_content_when_capture_content_is_false(self) -> None: + ctx, rec = _recording() + agents_mod = _fake_agents_module( + run=_make_run([{"output": _text_output("answer")}]) + ) + with ctx, _patched_agents(agents_mod): + await create_openai_agent_handler(capture_content=False)( + CONFIG, "my question", {}, {} + ) for span in rec.spans: assert [k for k in span.attributes if k.startswith("gen_ai.prompt")] == [] assert [ @@ -1030,14 +1047,14 @@ async def test_process_default_is_read_when_the_handler_runs(self) -> None: # Built before configure, with no capture flag. Vega's startup call # has to reach this handler when it later runs. handler = create_openai_agent_handler() - configure(options=SDKOptions(capture_content=True)) + configure(options=SDKOptions(capture_content=False)) ctx, rec = _recording() agents_mod = _fake_agents_module( run=_make_run([{"output": _text_output("answer")}]) ) with ctx, _patched_agents(agents_mod): await handler(CONFIG, "my question", {}, {}) - assert "my question" in json.dumps(rec.root.attributes) + assert "my question" not in json.dumps(rec.root.attributes) finally: _reset_for_testing() @@ -1676,7 +1693,7 @@ async def test_a_failed_turn_closes_the_open_text_message(self) -> None: assert len(ends) == 1 assert ends[0]["messageId"] == starts[0]["messageId"] - async def test_emits_no_content_by_default_on_the_streaming_path(self) -> None: + async def test_emits_content_by_default_on_the_streaming_path(self) -> None: turns = [{"deltas": ["hi"], "output": _text_output("hi")}] ctx, rec = _recording() agents_mod = _fake_agents_module(run_streamed=_make_run_streamed(turns, "hi")) @@ -1685,6 +1702,23 @@ async def test_emits_no_content_by_default_on_the_streaming_path(self) -> None: CONFIG, "q", {}, {} ): pass + assert any( + "gen_ai.input.messages" in span.attributes + or any(key.startswith("gen_ai.prompt") for key in span.attributes) + for span in rec.spans + ) + + async def test_emits_no_content_when_capture_is_false_on_the_streaming_path( + self, + ) -> None: + turns = [{"deltas": ["hi"], "output": _text_output("hi")}] + ctx, rec = _recording() + agents_mod = _fake_agents_module(run_streamed=_make_run_streamed(turns, "hi")) + with ctx, _patched_agents(agents_mod): + async for _ in await create_openai_agent_handler( + capture_content=False + ).stream(CONFIG, "q", {}, {}): + pass for span in rec.spans: assert [k for k in span.attributes if k.startswith("gen_ai.prompt")] == [] assert [n for n, _ in span.events if n.startswith("gen_ai.content")] == [] diff --git a/packages/openai-messages/agents.md b/packages/openai-messages/agents.md index 3057b39c..c99532f1 100644 --- a/packages/openai-messages/agents.md +++ b/packages/openai-messages/agents.md @@ -2,7 +2,7 @@ > **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with > tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler. -> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults). +> Conversation content is on unless the caller passes `capture_content=False`, or the process turns it off. See [Process-wide defaults](../client/README.md#process-wide-defaults). > `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code. diff --git a/packages/openai-messages/src/launchdarkly_ai_openai_messages/handler.py b/packages/openai-messages/src/launchdarkly_ai_openai_messages/handler.py index 5e804e41..3e95172a 100644 --- a/packages/openai-messages/src/launchdarkly_ai_openai_messages/handler.py +++ b/packages/openai-messages/src/launchdarkly_ai_openai_messages/handler.py @@ -386,9 +386,9 @@ def create_openai_messages_handler( Set *capture_content* to put prompts, model output, tool arguments and tool results on the emitted spans. ``None`` (the default) reads the process-wide default at invocation, - which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default - for this handler. Conversation content is PII, so a run emits only metadata, meaning - models, token counts, timings and tool names, until capture is turned on. + which is on until ``configure`` turns it off. ``True`` or ``False`` overrides that default + for this handler. Conversation content is PII. An explicit ``False`` emits only metadata, + meaning models, token counts, timings and tool names. """ report_usage("openai-messages.createOpenAIHandler", PACKAGE_NAME, __version__) return _create_openai_messages_handler(capture_content=capture_content) diff --git a/packages/openai-messages/tests/test_handler.py b/packages/openai-messages/tests/test_handler.py index b4bdb69a..ebed2248 100644 --- a/packages/openai-messages/tests/test_handler.py +++ b/packages/openai-messages/tests/test_handler.py @@ -690,7 +690,9 @@ async def test_carries_the_launchdarkly_attributes_and_feature_flag_event( assert attrs["launchdarkly.variation.key"] == "v" assert attrs["launchdarkly.run.id"] == "r" assert attrs["launchdarkly.graph.key"] == "g" - assert [n for n, _ in rec.root.events] == ["feature_flag"] + assert [ + name for name, _ in rec.root.events if not name.startswith("gen_ai.content") + ] == ["feature_flag"] async def test_child_spans_carry_no_launchdarkly_identity( self, mock_openai: MagicMock @@ -873,14 +875,28 @@ async def test_sets_status_ok_on_a_successful_turn( class TestContentCapture: """TELEMETRY-CONTRACT.md section 7.""" - async def test_emits_no_content_at_all_by_default( + async def test_emits_content_by_default(self, mock_openai: MagicMock) -> None: + ctx, rec = _recording() + from launchdarkly_ai_openai_messages import create_openai_messages_handler + + with ctx: + await create_openai_messages_handler()(CONFIG, "q", {}, {}) + assert any( + "gen_ai.input.messages" in span.attributes + or any(key.startswith("gen_ai.prompt") for key in span.attributes) + for span in rec.spans + ) + + async def test_emits_no_content_when_capture_content_is_false( self, mock_openai: MagicMock ) -> None: ctx, rec = _recording() from launchdarkly_ai_openai_messages import create_openai_messages_handler with ctx: - await create_openai_messages_handler()(CONFIG, "q", {}, {}) + await create_openai_messages_handler(capture_content=False)( + CONFIG, "q", {}, {} + ) for span in rec.spans: content_keys = [ k @@ -2315,7 +2331,7 @@ async def test_fails_the_spans_when_the_stream_raises( assert StatusCode.ERROR in rec.root.statuses assert rec.root.ended == 1 - async def test_emits_no_content_by_default_on_the_streaming_path( + async def test_emits_content_by_default_on_the_streaming_path( self, mock_openai: MagicMock ) -> None: self._patch_stream(mock_openai, ["hi"]) @@ -2325,6 +2341,20 @@ async def test_emits_no_content_by_default_on_the_streaming_path( with ctx: async for _ in await create_openai_messages_handler().stream(CONFIG, "q"): pass + assert any("gen_ai.input.messages" in span.attributes for span in rec.spans) + + async def test_emits_no_content_when_capture_is_false_on_the_streaming_path( + self, mock_openai: MagicMock + ) -> None: + self._patch_stream(mock_openai, ["hi"]) + ctx, rec = _recording() + from launchdarkly_ai_openai_messages import create_openai_messages_handler + + with ctx: + async for _ in await create_openai_messages_handler( + capture_content=False + ).stream(CONFIG, "q"): + pass for span in rec.spans: assert [k for k in span.attributes if k.startswith("gen_ai.prompt")] == [] assert [n for n, _ in span.events if n.startswith("gen_ai.content")] == [] diff --git a/packages/typesafe/tests/test_handler.py b/packages/typesafe/tests/test_handler.py index bdf66933..24fac6e8 100644 --- a/packages/typesafe/tests/test_handler.py +++ b/packages/typesafe/tests/test_handler.py @@ -41,7 +41,7 @@ def _client(response: object) -> MagicMock: async def test_provides_for_typesafe_messages() -> None: handler = create_typesafe_handler() assert handler.provides_for == ("TypeSafe", "messages") - assert handler.capture_content is False + assert handler.capture_content is True @pytest.mark.asyncio