diff --git a/weave.mdx b/weave.mdx index d222fa2e0c..95e1fa6054 100644 --- a/weave.mdx +++ b/weave.mdx @@ -53,7 +53,7 @@ W&B Weave provides Python and TypeScript libraries. To install the Weave library ```bash - pnpm install weave + npm install weave ``` diff --git a/weave/cookbooks/source/agents-quickstart.ipynb b/weave/cookbooks/source/agents-quickstart.ipynb index 3d7e76508d..7b0cb719e1 100644 --- a/weave/cookbooks/source/agents-quickstart.ipynb +++ b/weave/cookbooks/source/agents-quickstart.ipynb @@ -6,11 +6,11 @@ "source": [ "# Quickstart: Trace an agent\n", "\n", - "Trace a multi-turn agent with the Weave SDK. Sessions, turns, LLM calls, and tool calls render in the Agents view of your project.\n", + "Trace a multi-turn agent with the Weave SDK. Conversations, turns, LLM calls, and tool calls render in the Agents view of your project.\n", "\n", "The Weave SDK allows you to trace custom agents or agents created using popular SDKs. This quickstart guides you through how to manually integrate Weave into a custom-built multi-turn agent to emit and capture OpenTelemetry spans and render them in Weave's Agents view.\n", "\n", - "If you are looking to integrate Weave with popular SDKs or harnesses, such as the Claude Agents SDK or Codex, see the [Weave integration section](https://docs.wandb.ai/weave/guides/integrations). Weave autopatches into several popular agent-building SDKs and agent harnesses for quick integration.\n", + "If you are looking to integrate Weave with popular SDKs or harnesses, such as the Claude Agents SDK or Codex, see [Choose an agent integration](https://docs.wandb.ai/weave/agent-integration-quickstart). Weave autopatches into several popular agent-building SDKs and agent harnesses for quick integration.\n", "\n", "## What you'll learn\n", "\n", @@ -19,10 +19,11 @@ "This guide shows you how to:\n", "\n", "- Initialize Weave for agent tracing with `weave.init()`\n", - "- Open a session and a turn with `weave.start_session()` and `weave.start_turn()`\n", + "- Open a conversation and a turn with `weave.start_conversation()` and `weave.start_turn()`\n", "- Wrap LLM calls with `weave.start_llm()` and record usage\n", "- Wrap tool executions with `weave.start_tool()` and record results\n", - "- View the resulting session, turns, and tool calls in the Agents view\n", + "- Record complete token usage and a priceable model so token counts and cost render\n", + "- View the resulting conversation, turns, and tool calls in the Agents view\n", "\n", "## How the Weave SDK works with agents\n", "\n", @@ -30,7 +31,7 @@ "\n", "| Function | Maps to | OTel span |\n", "| --- | --- | --- |\n", - "| `weave.start_session(...)` | A conversation | (no span — groups turns) |\n", + "| `weave.start_conversation(...)` | A conversation | (no span — groups turns) |\n", "| `weave.start_turn(...)` | One user / agent exchange | `invoke_agent` |\n", "| `weave.start_llm(...)` | One LLM API call | `chat` |\n", "| `weave.start_tool(...)` | One tool execution | `execute_tool` |\n", @@ -141,7 +142,7 @@ "source": [ "## Run a traced multi-turn agent\n", "\n", - "The example below runs three turns in a single session. Each turn:\n", + "The example below runs three turns in a single conversation. Each turn:\n", "\n", "1. Opens a `chat` span and lets the LLM decide whether to call the tool\n", "2. If the LLM requested a tool, opens an `execute_tool` span around the call and feeds the result back to the LLM\n", @@ -170,9 +171,17 @@ " )\n", " msg = resp.choices[0].message\n", " llm.output(msg.content or \"\")\n", - " llm.usage = weave.Usage(\n", - " input_tokens=resp.usage.prompt_tokens,\n", - " output_tokens=resp.usage.completion_tokens,\n", + " # record() sets usage, the priced model, and response id in one call.\n", + " llm.record(\n", + " usage=weave.Usage(\n", + " input_tokens=resp.usage.prompt_tokens,\n", + " output_tokens=resp.usage.completion_tokens,\n", + " cache_read_input_tokens=getattr(\n", + " resp.usage.prompt_tokens_details, \"cached_tokens\", 0\n", + " ),\n", + " ),\n", + " response_id=resp.id,\n", + " response_model=resp.model,\n", " )\n", " history.append(msg.model_dump(exclude_none=True))\n", "\n", @@ -199,14 +208,21 @@ " resp = openai_client.chat.completions.create(model=MODEL, messages=history)\n", " msg = resp.choices[0].message\n", " llm.output(msg.content)\n", - " llm.usage = weave.Usage(\n", - " input_tokens=resp.usage.prompt_tokens,\n", - " output_tokens=resp.usage.completion_tokens,\n", + " llm.record(\n", + " usage=weave.Usage(\n", + " input_tokens=resp.usage.prompt_tokens,\n", + " output_tokens=resp.usage.completion_tokens,\n", + " cache_read_input_tokens=getattr(\n", + " resp.usage.prompt_tokens_details, \"cached_tokens\", 0\n", + " ),\n", + " ),\n", + " response_id=resp.id,\n", + " response_model=resp.model,\n", " )\n", " history.append({\"role\": \"assistant\", \"content\": msg.content})\n", " return msg.content\n", "\n", - "with weave.start_session(agent_name=\"research-bot\") as session:\n", + "with weave.start_conversation(agent_name=\"research-bot\") as conversation:\n", " history = []\n", " for question in [\n", " \"Who founded Anthropic?\",\n", @@ -217,6 +233,43 @@ " print(f\"AGENT: {run_turn(history, question)}\\n\")" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Record token usage and cost\n", + "\n", + "Each `chat` span carries token usage and a model id. Weave renders token counts from usage and derives cost from usage plus the model id, so an incomplete or unpriceable value shows `0 in / 0 out` tokens or `Cost -` even when the rest of the trace looks correct. `record(...)` sets these fields (along with `output_messages`, `response_id`, `reasoning`, and more) in one call. Only the fields you pass are applied.\n", + "\n", + "Two things must be right for cost to appear:\n", + "\n", + "- **Complete usage.** `input_tokens` is the *total* input, including any cached tokens. Weave prices cache reads and cache writes at their own rates and subtracts them from the input total, so `cache_read_input_tokens` and `cache_creation_input_tokens` must be reported *in addition to* a total `input_tokens` that includes them. For providers with prompt caching (for example, Anthropic), cached tokens routinely dominate the input, so omitting them makes usage and cost render as roughly zero.\n", + "- **A priceable model id.** Cost is a lookup on the model. Weave prefers `response_model` (the exact model the provider served) and falls back to the `model` you passed to `start_llm`. An alias such as `opus` or `sonnet` is not priceable and renders `Cost -`, so pass the concrete id the response returns (`resp.model`) as `response_model`.\n", + "\n", + "OpenAI counts cached tokens inside `prompt_tokens`, so the agent above maps directly. Anthropic reports cached tokens *separately* from `input_tokens`, so add them back into the total Weave prices against. The following snippet is illustrative — this notebook uses OpenAI, so there's no `anthropic_client` to run it against:\n", + "\n", + "```python\n", + "with weave.start_llm(model=MODEL, provider_name=\"anthropic\") as llm:\n", + " resp = anthropic_client.messages.create(\n", + " model=MODEL, max_tokens=1024, messages=history,\n", + " )\n", + " u = resp.usage\n", + " llm.output(resp.content[0].text)\n", + " llm.record(\n", + " usage=weave.Usage(\n", + " input_tokens=u.input_tokens\n", + " + u.cache_read_input_tokens\n", + " + u.cache_creation_input_tokens,\n", + " output_tokens=u.output_tokens,\n", + " cache_read_input_tokens=u.cache_read_input_tokens,\n", + " cache_creation_input_tokens=u.cache_creation_input_tokens,\n", + " ),\n", + " response_id=resp.id,\n", + " response_model=resp.model,\n", + " )\n", + "```" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -226,7 +279,7 @@ "When `weave.init()` runs, it prints a link to your project where you can see:\n", "\n", "- A row in the **Agents** tab for `research-bot`\n", - "- One session containing three turns\n", + "- One conversation containing three turns\n", "- Each turn (`invoke_agent`) with two `chat` spans and an `execute_tool` span nested inside\n", "- Token counts, latency, model, and the full message exchange on each `chat`\n", "\n", @@ -235,7 +288,7 @@ "## Next steps\n", "\n", "- Get a better understanding of how to [trace agents with Weave](https://docs.wandb.ai/weave/guides/tracking/trace-agents) and what features and options are available in the Weave SDK.\n", - "- See the [integration section](https://docs.wandb.ai/weave/guides/integrations) for more options on how to integrate Weave with your agents." + "- See [Choose an agent integration](https://docs.wandb.ai/weave/agent-integration-quickstart) for more options on how to integrate Weave with your agents." ] } ], diff --git a/weave/custom-agents-quickstart.mdx b/weave/custom-agents-quickstart.mdx index 41e3e2d419..29c2e28be6 100644 --- a/weave/custom-agents-quickstart.mdx +++ b/weave/custom-agents-quickstart.mdx @@ -8,7 +8,7 @@ keywords: ["OpenTelemetry", "OTel spans", "multi-turn agent", "tool calls", "tra The Weave SDK lets you trace agents built with popular SDKs or custom harnesses. This quickstart shows you how to manually integrate Weave into a custom-built multi-turn agent to emit and capture OpenTelemetry spans. For conceptual understanding about Weave for agents, see [Trace your agents](/weave/guides/tracking/trace-agents). -If you're looking to integrate Weave with SDKs or harnesses such as the Claude Agent SDK or Codex, see [Trace agent integrations](/weave/guides/tracking/trace-agent-integrations). Weave autopatches into several agent-building SDKs and agent harnesses for quick integration. +If you're looking to integrate Weave with SDKs or harnesses such as the Claude Agent SDK or Codex, see [Choose an agent integration](/weave/agent-integration-quickstart). Weave autopatches into several agent-building SDKs and agent harnesses for quick integration. ## What you'll learn @@ -22,6 +22,7 @@ This guide shows you how to: - Open a conversation and a turn with `start_conversation` / `startConversation` and `start_turn` / `startTurn`. - Wrap LLM calls with `start_llm` / `startLLM` and record usage. - Wrap tool executions with `start_tool` / `startTool` and record results. +- Record complete token usage and a priceable model so token counts and cost render. - View the resulting conversation, turns, and tool calls in the Agents view. ## How the Weave SDK works with agents @@ -169,7 +170,8 @@ The following example runs three turns in a single conversation. Each turn: 3. Opens a second `chat` span to produce the final answer. -```python lines highlight="9,11,17,29,42,46,53" Python +```python lines highlight="10,12,19,38,51,55,69" Python +import weave from openai import OpenAI openai_client = OpenAI() @@ -186,9 +188,17 @@ def run_turn(history, user_message): ) msg = resp.choices[0].message llm.output(msg.content or "") - llm.usage = weave.Usage( - input_tokens=resp.usage.prompt_tokens, - output_tokens=resp.usage.completion_tokens, + # record() sets usage, the priced model, and response id in one call. + llm.record( + usage=weave.Usage( + input_tokens=resp.usage.prompt_tokens, + output_tokens=resp.usage.completion_tokens, + cache_read_input_tokens=getattr( + resp.usage.prompt_tokens_details, "cached_tokens", 0 + ), + ), + response_id=resp.id, + response_model=resp.model, ) history.append(msg.model_dump(exclude_none=True)) @@ -215,9 +225,16 @@ def run_turn(history, user_message): resp = openai_client.chat.completions.create(model=MODEL, messages=history) msg = resp.choices[0].message llm.output(msg.content) - llm.usage = weave.Usage( - input_tokens=resp.usage.prompt_tokens, - output_tokens=resp.usage.completion_tokens, + llm.record( + usage=weave.Usage( + input_tokens=resp.usage.prompt_tokens, + output_tokens=resp.usage.completion_tokens, + cache_read_input_tokens=getattr( + resp.usage.prompt_tokens_details, "cached_tokens", 0 + ), + ), + response_id=resp.id, + response_model=resp.model, ) history.append({"role": "assistant", "content": msg.content}) return msg.content @@ -233,8 +250,9 @@ with weave.start_conversation(agent_name="research-bot") as conversation: print(f"AGENT: {run_turn(history, question)}\n") ``` -```typescript lines highlight="10,13,40,55,77" TypeScript twoslash +```typescript lines highlight="11,14,25,47,62,70,89" // @noErrors +import * as weave from 'weave'; import OpenAI from 'openai'; const openaiClient = new OpenAI(); @@ -257,10 +275,16 @@ async function runTurn(history: any[], userMessage: string): Promise +## Record token usage and cost + +Each `chat` span carries token usage and a model id. Weave renders token counts from usage and derives cost from usage plus the model id, so an incomplete or unpriceable value shows `0 in / 0 out` tokens or `Cost -` even when the rest of the trace looks correct. `record(...)` sets these fields (along with `output_messages`, `response_id`, `reasoning`, and more) in one call. Only the fields you pass are applied. + +Two things must be right for cost to appear: + +- **Complete usage.** `input_tokens` is the *total* input, including any cached tokens. Weave prices cache reads and cache writes at their own rates and subtracts them from the input total, so `cache_read_input_tokens` and `cache_creation_input_tokens` must be reported *in addition to* a total `input_tokens` that includes them. For providers with prompt caching (for example, Anthropic), cached tokens routinely dominate the input, so omitting them makes usage and cost render as roughly zero. +- **A priceable model id.** Cost is a lookup on the model. Weave prefers `response_model` (the exact model the provider served) and falls back to the `model` you passed to `start_llm`. An alias such as `opus` or `sonnet` is not priceable and renders `Cost -`, so pass the concrete id the response returns (`resp.model`) as `response_model`. + +OpenAI counts cached tokens inside `prompt_tokens`, so the example above maps directly. Anthropic reports cached tokens *separately* from `input_tokens`, so add them back into the total Weave prices against: + + +```python lines Python +with weave.start_llm(model=MODEL, provider_name="anthropic") as llm: + resp = anthropic_client.messages.create( + model=MODEL, max_tokens=1024, messages=history, + ) + u = resp.usage + llm.output(resp.content[0].text) + llm.record( + usage=weave.Usage( + input_tokens=u.input_tokens + + u.cache_read_input_tokens + + u.cache_creation_input_tokens, + output_tokens=u.output_tokens, + cache_read_input_tokens=u.cache_read_input_tokens, + cache_creation_input_tokens=u.cache_creation_input_tokens, + ), + response_id=resp.id, + response_model=resp.model, + ) +``` + +```typescript lines TypeScript twoslash +// @noErrors +const u = resp.usage; +llm.record({ + usage: { + inputTokens: + u.input_tokens + u.cache_read_input_tokens + u.cache_creation_input_tokens, + outputTokens: u.output_tokens, + cacheReadInputTokens: u.cache_read_input_tokens, + cacheCreationInputTokens: u.cache_creation_input_tokens, + }, + responseId: resp.id, + responseModel: resp.model, +}); +``` + + ## See your agent traces in the Agents view When `weave.init()` runs, it prints a link to your project where you can see: @@ -339,7 +418,30 @@ When `weave.init()` runs, it prints a link to your project where you can see: Click any turn to inspect the inputs, outputs, tool arguments, and tool results. +## Link to a conversation from your app + +To deep-link from your own UI to a conversation in the Weave Agents view, build the URL from your entity, project, and the conversation id. `weave.init()` returns a client that carries `entity` and `project`, and `start_conversation` exposes `conversation_id`. + + +```python lines Python +from weave.trace.urls import agent_conversation_path + +client = weave.init(f"{TEAM}/{PROJECT}") + +with weave.start_conversation(agent_name="research-bot") as conversation: + # ... run turns ... + url = agent_conversation_path( + client.entity, client.project, conversation.conversation_id + ) + print(f"View this conversation at {url}") +``` + +```plaintext TypeScript lines +This feature is not available in TypeScript. +``` + + ## Next steps - Learn how to [trace agents with Weave](/weave/guides/tracking/trace-agents) and what features and options are available in the Weave SDK. -- See [Trace agent integrations](/weave/guides/tracking/trace-agent-integrations) for more options about how to integrate Weave with your agents. +- See [Choose an agent integration](/weave/agent-integration-quickstart) for more options about how to integrate Weave with your agents.