Skip to content

feat(sdk): TokenJamClient LiteLLM Integration#61

Merged
anilmurty merged 1 commit into
mainfrom
feat/tokenjam-client-for-litellm
May 12, 2026
Merged

feat(sdk): TokenJamClient LiteLLM Integration#61
anilmurty merged 1 commit into
mainfrom
feat/tokenjam-client-for-litellm

Conversation

@anilmurty
Copy link
Copy Markdown
Contributor

Summary

Adds tokenjam.sdk.TokenJamClient — a thin HTTP client that POSTs a single LiteLLM call as an OTLP JSON span to a running tj serve. Unblocks the upstream BerriAI/litellm named-callback PR (litellm.success_callback = [\"tokenjam\"]), which depends on a public surface that doesn't require running inside a tokenjam-aware app.

Bumps version to 0.2.2.

Public surface

```python
from tokenjam.sdk import TokenJamClient

c = TokenJamClient(
endpoint="http://localhost:7391\", # default
ingest_secret="...", # optional Bearer auth
)
c.emit_litellm_span(
kwargs={"model": "openai/gpt-4o-mini", "metadata": {"tj_agent_id": "my-agent"}},
response_obj=response,
start_time=start,
end_time=end,
success=True,
)
```

What gets captured

  • gen_ai.request.model (provider prefix stripped — `openai/gpt-4o-mini` → `gpt-4o-mini`)
  • gen_ai.provider.name (from response._hidden_params['custom_llm_provider'], falling back to the model prefix)
  • gen_ai.usage.input_tokens / output_tokens / cache_read_tokens / cache_creation_tokens
  • tokenjam.cost_usd (from kwargs['response_cost'] or response._hidden_params['response_cost'])
  • gen_ai.agent.id / gen_ai.conversation.id (from kwargs['metadata']['tj_agent_id' | 'tj_session_id'])
  • Status code (OK / ERROR) and message on failure

Design notes

  • Non-blocking: every error path is caught and logged at debug — the method never raises into the caller's request path. Matches the idiom expected by LiteLLM, Langfuse, Helicone, etc.
  • No SDK initialization required: the client is a plain HTTP POSTer with no dependency on bootstrap/ensure_initialised() or the OTel TracerProvider, so it can be safely instantiated from inside LiteLLM without side effects on the host process.
  • Endpoint normalization: accepts either the server base URL or the full /api/v1/spans path.
  • For in-process tokenjam users, patch_litellm() remains the preferred path — it produces equivalent spans via the OTel pipeline.

Files

  • tokenjam/sdk/client.py (new, 230 lines) — TokenJamClient + pure-function payload builders
  • tokenjam/sdk/__init__.py — re-export TokenJamClient
  • tests/unit/test_litellm_client.py (new, 10 tests) — covers payload shape, provider/model fallbacks, dict vs pydantic usage, error status, auth header, swallowed connection / build errors
  • pyproject.toml, sdk-ts/package.json, sdk-ts/package-lock.json — bump to 0.2.2

Test plan

  • `pytest tests/unit/ tests/synthetic/` — 316 passed
  • `ruff check` clean on new files
  • `mypy tokenjam/sdk/client.py` clean
  • CI green
  • After merge: create v0.2.2 GitHub release to trigger PyPI publish
  • Then proceed with upstream BerriAI/litellm PR

🤖 Generated with Claude Code

Adds tokenjam.sdk.TokenJamClient — a thin HTTP client that POSTs a single
LiteLLM call as an OTLP JSON span to a running tj serve. Designed to be
embedded in foreign codebases that cannot rely on the in-process OTel
TracerProvider — most notably the upstream BerriAI/litellm named-callback
machinery (litellm.success_callback = ["tokenjam"]).

Public surface:
    from tokenjam.sdk import TokenJamClient
    c = TokenJamClient(endpoint="http://localhost:7391", ingest_secret=...)
    c.emit_litellm_span(kwargs, response_obj, start_time, end_time, success)

The method translates LiteLLM's callback payload (kwargs/response_obj/times)
into an OTLP JSON span — provider, model, input/output tokens, cache tokens,
cost (from kwargs["response_cost"] or response._hidden_params), and optional
agent/session tags from kwargs["metadata"]["tj_agent_id" | "tj_session_id"].

Non-blocking by design: all errors are logged at debug and the event is
dropped — never propagates an exception into the caller's request path.

Bumps version to 0.2.2.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@anilmurty anilmurty changed the title feat(sdk): TokenJamClient for external integrations (LiteLLM upstream prereq) feat(sdk): TokenJamClient LiteLLM Integration May 12, 2026
@anilmurty anilmurty merged commit 432dd8f into main May 12, 2026
4 checks passed
@anilmurty anilmurty deleted the feat/tokenjam-client-for-litellm branch May 12, 2026 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant