Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ The client manages a singleton connection to LaunchDarkly and the associated tel
|---|---|
| `init_client(options?)` | Auto-discovers and initializes `launchdarkly-server-sdk` (optional dep, loaded via `importlib`). Optional — the first AI API call triggers lazy init when `LD_SDK_KEY` is set. Accepts optional overrides for SDK key, base URIs, service name, environment, and OTLP endpoint. Returns `Awaitable[LDClientInterface]`. On every successful path, including the already-initialized path, flushes `$ld:ai:sdk:info` for any LaunchDarkly AI packages that have not yet reported. Does not take behavior defaults. |
| `init_client(client=...)` | **BYOC overload** — accepts a pre-initialized `LDClientInterface`. Stores it directly without calling the SDK. Flushes pending `$ld:ai:sdk:info` events. |
| `configure(*, options)` | Sets process-wide behavior defaults on the lifecycle singleton. `options` is a frozen `SDKOptions` (`capture_content: bool = False`). Keyword-only. Handlers read it at invocation. An explicit factory or wrapper `capture_content` overrides it. Replaceable until the first AI invocation; an identical value is allowed after. Not read from request data. |
| `configure(*, options)` | Sets process-wide behavior defaults on the lifecycle singleton. `options` is a frozen `SDKOptions` (`capture_content: bool = True`). Keyword-only. Handlers read it at invocation. An explicit factory or wrapper `capture_content` overrides it, and an explicit `False` strips content everywhere, including offline evaluation spans. Replaceable until the first AI invocation; an identical value is allowed after. Not read from request data. |
| `get_client()` | Returns the initialized `LDClientInterface`. Throws if initialization has not completed. |
| `shutdown()` | Flushes all pending events and telemetry, then closes the client. Must be awaited before the process exits. Clears sdk-info reporting so a later client reports again. Restores the built-in SDK defaults. |
| `register_ai_sdk_package(name, version)` | Records a LaunchDarkly AI package identity. Handler and convenience packages call this at import time. |
Expand Down Expand Up @@ -571,14 +571,15 @@ beside the input count, so its handlers add it in. OpenAI and LangChain already
input, so theirs pass the figure through. Centralising that rule would double-count for two
providers out of three. `SpanUsage` is the type that means the folding is already done.

#### Content is off by default
#### Content is on by default

Every factory takes `capture_content: bool | None = None`. `None` reads `configure()` at
invocation, which is off until a process sets `SDKOptions(capture_content=True)`. An explicit
`True` or `False` overrides that default. Guard at the call site as well as inside the helper: the
helper's guard makes a forgotten call site harmless, and the call site's guard avoids serialising
JSON that would then be discarded, once per turn, in a loop. Resolve the bool when the handler
runs, not when it is constructed.
invocation. The built-in default is on until a process sets `SDKOptions(capture_content=False)`.
An explicit `True` or `False` overrides that default, and an explicit `False` strips content
everywhere, including offline evaluation spans. Guard at the call site as well as inside the
helper: the helper's guard makes a forgotten call site harmless, and the call site's guard avoids
serialising JSON that would then be discarded, once per turn, in a loop. Resolve the bool when the
handler runs, not when it is constructed.

#### Finish reasons have three mechanisms, not one

Expand Down
15 changes: 9 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -542,20 +542,23 @@ This matters most on Anthropic, which reports cache reads and writes beside the
than inside it. A turn that reads 19,971 tokens from cache and writes 3,580 more reports an input of
3 from the provider; the span reports 23,554.

### Conversation content is off by default
### Conversation content is on by default

Prompts, model output, tool arguments and tool results are personal data, so a span carries only
metadata unless you ask for more: models, token counts, timings, tool names.
Prompts, model output, tool arguments and tool results are personal data. The built-in default
sends that text. A span carries only metadata — models, token counts, timings, tool names — when
capture is turned off.

Pass `capture_content=True` to a handler factory to include the conversation, or turn it on for every handler from process startup. See [Process-wide defaults](packages/client/README.md#process-wide-defaults).
Pass `capture_content=False` to a handler factory to leave the conversation off that handler, or
turn it off for every handler from process startup. An explicit `False` wins everywhere, including
on offline evaluation spans. See [Process-wide defaults](packages/client/README.md#process-wide-defaults).

```python
from launchdarkly_ai_claude_messages import create_claude_messages_handler

handler = create_claude_messages_handler(capture_content=True)
handler = create_claude_messages_handler(capture_content=False)
```

Turning this on sends the text of every request and response to whatever collector the SDK points
Leaving this on sends the text of every request and response to whatever collector the SDK points
at.

### Streaming
Expand Down
13 changes: 7 additions & 6 deletions TELEMETRY-CONTRACT.md
Original file line number Diff line number Diff line change
Expand Up @@ -397,12 +397,13 @@ error for an abandoned stream, so two dashboards would describe the same run dif
Every handler factory takes `capture_content: bool | None = None`.

`None` reads the process-wide default at invocation (`configure(options=SDKOptions(...))`), which
is off. `True` or `False` overrides that default for the handler, and an explicit `False` wins
over a configured `True`. A convenience wrapper's `capture_content` is the per-call override: when
the caller passes it, that value is what the handler is built with; when they omit it, the wrapper
passes `None` and the process default applies. Conversation content is personal data. A run emits
only metadata, meaning models, token counts, timings and tool names, until capture is turned on.
Turning it on sends the text of every request and response to whatever collector the SDK points at.
is on. `True` or `False` overrides that default for the handler, and an explicit `False` wins
over a configured `True`, including on offline evaluation spans. A convenience wrapper's
`capture_content` is the per-call override: when the caller passes it, that value is what the
handler is built with; when they omit it, the wrapper passes `None` and the process default
applies. Conversation content is personal data. The default sends the text of every request and
response to whatever collector the SDK points at. An explicit `False` emits only metadata, meaning
models, token counts, timings and tool names.

Guard twice, on purpose. The helper returns without writing when capture is off, which makes a
forgotten call site harmless. The call site also checks before building the argument, which avoids
Expand Down
16 changes: 9 additions & 7 deletions examples/capture_content.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
"""
Example: enable conversation content for every handler at process startup.

The handlers in the global registry are built with no ``capture_content``
argument. They read the process default when they run. ``configure`` is
separate from ``init_client``: ``main.py`` initializes the client, and this
example calls ``configure`` once before the first AI invocation. An explicit
``True`` or ``False`` on a factory would override this default.
Example: pin conversation content on for every handler at process startup.

The built-in default is already on. The handlers in the global registry are
built with no ``capture_content`` argument and read that default when they
run. ``configure`` is separate from ``init_client``: ``main.py`` initializes
the client, and this example calls ``configure`` once before the first AI
invocation so the choice is visible at startup. Pass
``SDKOptions(capture_content=False)`` to opt out. An explicit ``False`` on a
factory removes the transcript everywhere, including offline evaluation spans.

Usage (via main.py):
python main.py capture-content <flag-key> "<user input>"
Expand Down
8 changes: 4 additions & 4 deletions examples/conversation.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@
the same id: root, ``chat``, ``execute_tool``, and the judge's own ``invoke_agent``.
2. Each turn shows a score badge, sourced from the ``gen_ai.evaluation.result`` span event on
the judge span.
3. No judge reasoning anywhere in the telemetry. The score and the judge's config key are
exported; the explanation is not, because it is model prose about the user's conversation
and content attributes require ``capture_content``. The reasoning IS printed below, straight
from ``judge_results`` — that is the caller's copy, and it is unaffected.
3. The judge explanation is on the span, because capture is on by default. The score and
the judge's config key are exported either way. An explicit ``False`` removes the
explanation from telemetry. The reasoning IS printed below, straight from
``judge_results`` — that is the caller's copy, and it is unaffected.

The flag key must point at an AI Config with a ``judge_configuration``, otherwise there are no
judge turns to look at.
Expand Down
2 changes: 1 addition & 1 deletion packages/claude-agents/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ asyncio.run(main())

| Option | Default | Description |
|---|---|---|
| `capture_content` | `None` | Put prompts, model output, and tool arguments and results on the emitted spans. `None` follows `configure()`, which is off by default. |
| `capture_content` | `None` | Put prompts, model output, and tool arguments and results on the emitted spans. `None` follows `configure()`, which is on by default. An explicit `False` leaves them off. |
| `cwd` | the process's working directory | Working directory for the Claude Code process the Agent SDK starts. A `str` or path-like. |
| `setting_sources` | `[]` (none) | Which Claude Code settings sources to load: `"user"` (`~/.claude/settings.json`), `"project"` (`.claude/settings.json`, and `CLAUDE.md`), `"local"` (`.claude/settings.local.json`). |

Expand Down
2 changes: 1 addition & 1 deletion packages/claude-agents/agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with
> tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler.
> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults).
> Conversation content is on unless the caller passes `capture_content=False`, or the process turns it off. See [Process-wide defaults](../client/README.md#process-wide-defaults).
> `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code.


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -711,8 +711,8 @@ def create_claude_agents_handler(

Set *capture_content* to put prompts, model output, tool arguments and tool results on the
emitted spans. ``None`` (the default) reads the process-wide default at invocation,
which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default
for this handler. See TELEMETRY-CONTRACT.md section 7.
which is on until ``configure`` turns it off. ``True`` or ``False`` overrides that default
for this handler. An explicit ``False`` emits only metadata. See TELEMETRY-CONTRACT.md section 7.

*cwd* and *setting_sources* configure the Claude Code process the Agent SDK starts, not the
model, so they come from the application and never from a config. Each one given is passed to
Expand Down
34 changes: 28 additions & 6 deletions packages/claude-agents/tests/test_handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -862,7 +862,25 @@ async def test_cache_tokens_folded_into_input(


class TestContentCapture:
async def test_emits_no_content_at_all_by_default(
async def test_emits_content_by_default(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(
handler_mod,
"query",
_fake_query(
[
assistant_message(content=[TextBlock(text="hi")]),
result_message("hi"),
]
),
)
await create_claude_agents_handler()(BASE_CONFIG, "q")
[chat] = named("chat ")
assert "gen_ai.input.messages" in chat.attributes
assert "gen_ai.output.messages" in chat.attributes

async def test_emits_no_content_when_capture_content_is_false(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(
Expand All @@ -872,22 +890,22 @@ async def test_emits_no_content_at_all_by_default(
[init_message(tools=["Read"]), assistant_message(), result_message()]
),
)
await create_claude_agents_handler()(TOOL_CONFIG, "q")
await create_claude_agents_handler(capture_content=False)(TOOL_CONFIG, "q")
for span in (root(), *named("chat ")):
assert "gen_ai.input.messages" not in span.attributes
assert "gen_ai.output.messages" not in span.attributes
assert "gen_ai.system_instructions" not in span.attributes
assert "gen_ai.tool.definitions" not in span.attributes

async def test_root_carries_no_content_by_default_either(
async def test_root_carries_no_output_when_capture_is_false(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(
handler_mod,
"query",
_fake_query([assistant_message(), result_message("hi")]),
)
await create_claude_agents_handler()(BASE_CONFIG, "q")
await create_claude_agents_handler(capture_content=False)(BASE_CONFIG, "q")
assert "gen_ai.output.messages" not in root().attributes

async def test_tool_call_arguments_and_result_gated_by_capture(
Expand Down Expand Up @@ -925,7 +943,7 @@ async def _query(**kwargs: Any) -> AsyncIterator[Any]:
yield result_message("done")

monkeypatch.setattr(handler_mod, "query", _query)
await create_claude_agents_handler()(
await create_claude_agents_handler(capture_content=False)(
TOOL_CONFIG, "q", {"search": lambda _: "r"}
)
[tool_span] = named("execute_tool ")
Expand Down Expand Up @@ -1610,7 +1628,11 @@ async def test_ld_span_attributes_land_on_root_only(
assert attrs["launchdarkly.config.key"] == "cfg"
assert attrs["launchdarkly.variation.key"] == "var"
assert attrs["launchdarkly.run.id"] == "run-1"
assert [e.name for e in root().events] == ["feature_flag"]
assert [
event.name
for event in root().events
if not event.name.startswith("gen_ai.content")
] == ["feature_flag"]


class TestModelParametersForwarding:
Expand Down
2 changes: 1 addition & 1 deletion packages/claude-messages/agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with
> tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler.
> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults).
> Conversation content is on unless the caller passes `capture_content=False`, or the process turns it off. See [Process-wide defaults](../client/README.md#process-wide-defaults).
> `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code.


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -537,9 +537,9 @@ def create_claude_messages_handler(

Set *capture_content* to put prompts, model output, tool arguments and tool results on the
emitted spans. ``None`` (the default) reads the process-wide default at invocation,
which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default
for this handler. Conversation content is PII, so a run emits only metadata, meaning
models, token counts, timings and tool names, until capture is turned on. Turning this
which is on until ``configure`` turns it off. ``True`` or ``False`` overrides that default
for this handler. Conversation content is PII. An explicit ``False`` emits only metadata,
meaning models, token counts, timings and tool names. Turning this
on sends the text of every request and response to whatever collector the SDK is pointed at.
"""
report_usage(
Expand Down
40 changes: 35 additions & 5 deletions packages/claude-messages/tests/test_handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -1007,7 +1007,9 @@ async def test_carries_the_launchdarkly_attributes_and_feature_flag_event(
assert attrs["launchdarkly.variation.key"] == "v"
assert attrs["launchdarkly.run.id"] == "r"
assert attrs["launchdarkly.graph.key"] == "g"
assert [n for n, _ in rec.root.events] == ["feature_flag"]
assert [
name for name, _ in rec.root.events if not name.startswith("gen_ai.content")
] == ["feature_flag"]

async def test_child_spans_carry_no_launchdarkly_identity(
self, mock_anthropic: MagicMock
Expand Down Expand Up @@ -1188,15 +1190,29 @@ async def test_sets_status_ok_on_a_successful_turn(
class TestContentCapture:
"""TELEMETRY-CONTRACT.md section 7."""

async def test_emits_no_content_at_all_by_default(
async def test_emits_content_by_default(self, mock_anthropic: MagicMock) -> None:
ctx, rec = _recording()
from launchdarkly_ai_claude_messages import create_claude_messages_handler

with ctx:
await create_claude_messages_handler()(CONFIG, "q", {}, {})
assert any(
"gen_ai.input.messages" in span.attributes
or any(key.startswith("gen_ai.prompt") for key in span.attributes)
for span in rec.spans
)

async def test_emits_no_content_when_capture_content_is_false(
self, mock_anthropic: MagicMock
) -> None:
# Conversation content is PII. This is the assertion worth pinning hardest.
# An explicit false still strips conversation content. This is the opt-out.
ctx, rec = _recording()
from launchdarkly_ai_claude_messages import create_claude_messages_handler

with ctx:
await create_claude_messages_handler()(CONFIG, "q", {}, {})
await create_claude_messages_handler(capture_content=False)(
CONFIG, "q", {}, {}
)
for span in rec.spans:
content_keys = [
k
Expand Down Expand Up @@ -2089,7 +2105,7 @@ async def test_fails_the_spans_when_the_stream_raises(
assert StatusCode.ERROR in rec.root.statuses
assert rec.root.ended == 1

async def test_emits_no_content_by_default_on_the_streaming_path(
async def test_emits_content_by_default_on_the_streaming_path(
self, mock_anthropic: MagicMock
) -> None:
self._patch_stream(mock_anthropic, ["hi"])
Expand All @@ -2099,6 +2115,20 @@ async def test_emits_no_content_by_default_on_the_streaming_path(
with ctx:
async for _ in await create_claude_messages_handler().stream(CONFIG, "q"):
pass
assert any("gen_ai.input.messages" in span.attributes for span in rec.spans)

async def test_emits_no_content_when_capture_is_false_on_the_streaming_path(
self, mock_anthropic: MagicMock
) -> None:
self._patch_stream(mock_anthropic, ["hi"])
ctx, rec = _recording()
from launchdarkly_ai_claude_messages import create_claude_messages_handler

with ctx:
async for _ in await create_claude_messages_handler(
capture_content=False
).stream(CONFIG, "q"):
pass
for span in rec.spans:
assert [k for k in span.attributes if k.startswith("gen_ai.prompt")] == []
assert [n for n, _ in span.events if n.startswith("gen_ai.content")] == []
Expand Down
Loading
Loading