RFC: Proposing a vendor-neutral gen_ai.* semantic convention to complement OpenLLMetry — seeking feedback from maintainers

[sauravGit]

Hi OpenLLMetry / Traceloop community,

I saw that OpenLLMetry's semantic conventions are now part of the OpenTelemetry project — that's a big step and directly inspired this proposal.

I'm working on a vendor-neutral, OTEL-compatible semantic convention and SDK layer for standardizing LLM observability metrics across providers and frameworks: https://github.com/sauravGit/open-llm-observability

The core problem: Even with OpenLLMetry doing great work on instrumentation, teams instrumenting multi-provider or multi-framework LLM apps still hit friction at the metric layer — different names for latency, tokens, cost, etc. depending on which tool or SDK they're using.

What I'm proposing: A canonical gen_ai.* metric schema that:

• Standardizes core KPI names: gen_ai.latency, gen_ai.time_to_first_token, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cost, gen_ai.error_rate
• Aligns with OTEL GenAI span conventions (gen_ai.system, gen_ai.request.model, gen_ai.operation.name)
• Defines interoperability rules so different instrumentation libraries (including OpenLLMetry) can map to the same canonical schema
• Includes a Python SDK scaffold emitting OTEL-native metrics

Why I'm posting here specifically: OpenLLMetry is the most widely-deployed OTEL-based LLM instrumentation library. If this convention is going to be useful in practice, it needs to be compatible with (and ideally aligned with) what OpenLLMetry already emits.

Questions for maintainers:

1. What metric names does OpenLLMetry currently emit for latency, TTFT, and token counts? Are they already in gen_ai.* namespace?
2. Are there any plans in the OpenTelemetry GenAI SIG to formalize a metric naming convention alongside the span conventions?
3. Would the OpenLLMetry team be open to collaborating on a shared metric convention layer?

Links:

• Full RFC: https://github.com/sauravGit/open-llm-observability/blob/main/RFC.md
• RFC discussion: RFC: Universal LLM Observability Semantic Convention (v0.4) sauravGit/open-llm-observability#1
• OTel spec discussion I opened: RFC feedback request: Vendor-neutral LLM observability semantic convention (gen_ai.* metrics) — looking for input from GenAI SIG maintainers open-telemetry/opentelemetry-specification#5069

This is explicitly v0.1 and designed to evolve. Happy to align naming and structure with whatever the OTel GenAI SIG and OpenLLMetry settle on.

[sauravGit]

Update: RFC v0.2 now available — with OpenLLMetry migration table

Since posting this, the RFC has been significantly expanded. Key additions most relevant to OpenLLMetry:

Corrected canonical names (aligned with the OTel GenAI SIG direction):

• gen_ai.client.operation.duration — Histogram, s
• gen_ai.client.time_to_first_token — Histogram, s
• gen_ai.usage.input_tokens — Counter, {token}
• gen_ai.usage.output_tokens — Counter, {token}
• gen_ai.usage.cost — Counter, usd

Migration shim table for OpenLLMetry specifically:

OpenLLMetry metric	Canonical name
llm.request.duration	gen_ai.client.operation.duration
llm.usage.prompt_tokens	gen_ai.usage.input_tokens
llm.usage.completion_tokens	gen_ai.usage.output_tokens
llm.usage.total_cost	gen_ai.usage.cost
llm.time_to_first_token	gen_ai.client.time_to_first_token

A Python migration shim pattern is included in the RFC that lets existing OpenLLMetry instrumentation re-emit data under canonical names without changing call sites.

Also new:

• Extension packs: gen_ai.rag.*, gen_ai.agent.*, gen_ai.eval.*, gen_ai.safety.*
• An OTel maintainer (@trask) engaged and pointed to semantic-conventions-genai as the right upstream venue — will be opening an issue there
• Working SDK examples (Python + TypeScript), PromQL dashboards, versioning policy

Full RFC v0.2: https://github.com/sauravGit/open-llm-observability/blob/main/RFC.md

Would especially value OpenLLMetry maintainer input on whether the migration shim approach is something the project would want to adopt officially.

[sauravGit]

RFC v0.3 update — three alignment fixes based on OTel upstream research:

1. Reframed as normalization/migration layer on top of existing OTel GenAI conventions (gen_ai.client.operation.duration, gen_ai.server.time_to_first_token, gen_ai.client.token.usage) — not a new parallel standard
2. OpenLLMetry → canonical migration table updated with correct mappings: llm.request.duration → gen_ai.client.operation.duration (convert ms→s), gen_ai.usage.prompt_tokens → gen_ai.usage.input_tokens, gen_ai.usage.completion_tokens → gen_ai.usage.output_tokens
3. gen_ai.usage.cost moved to optional Cost extension pack — not Core REQUIRED

Formal upstream proposal is now open at semantic-conventions-genai #101, consolidating #14, #23, #76, #93.

Full RFC v0.3: https://github.com/sauravGit/open-llm-observability/blob/main/RFC.md

Would especially value OpenLLMetry maintainer input on whether the migration shim approach is something the project would want to officially support.

[richardchen874-sys]

Vendor-neutral gen_ai telemetry conventions should preserve raw provider usage beside normalized totals, because optimization and reconciliation answer different questions.

I would keep provider, model, request type, prompt/output tokens, cached-token behavior, latency, retry count, and final billing bucket together in one event. That makes cost debugging and future routing decisions much easier.

This overlaps with something I am testing: an OpenAI-compatible multi-model API layer around official Chinese models, especially around cost-aware routing and normalized usage behavior. For openllmetry, would the first win be user-facing cost transparency, internal spend debugging, or routing cheaper calls to cheaper models?