feat(semconv): executable OTel GenAI semantic-convention contract - #4397
Open
galkleinman wants to merge 40 commits into
Open
feat(semconv): executable OTel GenAI semantic-convention contract#4397galkleinman wants to merge 40 commits into
galkleinman wants to merge 40 commits into
Conversation
Design for agent-maintained issue resolution: weaver-generated semconv conformance contract as ground truth, layered CLAUDE.md/skills/lessons substrate, and a mechanically enforced RED->GREEN TDD gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pin sweep rate limit, define tier-1 criteria, specify recording manifest location and generated-contract commit policy, and clarify that the sweep is a separate write scope rather than a fixer path-guard exception. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…wledge AGENTS.md becomes canonical with CLAUDE.md as a symlink, matching the contract repo's own layout. Procedures move to plain markdown with thin .claude/skills wrappers holding no content. Accumulated lessons and observed SDK behaviour become an OKF v0.2 bundle, whose generated/verified trust tiers and stale_after fields replace hand-rolled equivalents. Adds per-role path guards and a runner indirection so orchestration is not hard-coded to one agent vendor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight tasks: vendored registry with pinned SHA, weaver codegen to a committed contract module, conformance harness, declared-extensions registry, and warn-only rollout across in-scope packages. Weaver pipeline verified end-to-end before writing: resolve produces 18 span groups against the pinned ref via the otel/weaver container. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Move the contract generator into the semconv package (the repo has no root-level Python project, so root tests would never run in CI), and replace verbatim per-package test copies with a shared helper mirroring the existing _testing.py precedent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the Task 3 generator (_generator.py) that converts weaver's resolved GenAI semconv registry JSON into the committed, importable SPANS contract (generated.py) consumed by the Task 4 conformance harness. Also adds an E501 per-file-ignore for generated.py in ruff config since its rendered attribute/enum literals are long by construction and must never be hand-wrapped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Makefile FILTERED path resolves relative to weaver CWD, not the registry dir. Missing requirement_level now fails loudly instead of defaulting to recommended -- weaver back-fills, so the default was unreachable and would have silently downgraded a required attribute to non-blocking. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…espect open enums Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Thread `expected` through the shared test helper, and declare anthropic's expected attribute set including gen_ai.response.finish_reasons -- the declaration that makes #4362 detectable. Mark the superseded Task 4 code block as amended. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mirrors the existing _testing.py precedent: check_exported_spans/gen_ai_spans in one place so per-package conformance test files hold no duplicated check logic (Task 6/7 groundwork).
Wires the conformance harness into anthropic's test suite in warn-only mode (ENFORCING = False), reusing existing VCR cassettes from test_messages.py and test_structured_outputs.py rather than recording new ones. Adds a tool.uv.sources override so the package installs the local (unpublished) semantic-conventions-ai source instead of the stale PyPI 0.5.1 wheel, which lacks the conformance module. Proved the harness is not vacuous: flipping ENFORCING to True fails test_structured_output_span_conforms on a real undeclared_gen_ai violation (gen_ai.request.structured_output_schema, set in span_utils.py for output_format requests but not declared by the contract or extensions.py). The messages/streaming legacy-path spans conform cleanly against these cassettes.
Wires the conformance harness into ollama's test suite in warn-only mode (ENFORCING = False), reusing existing VCR cassettes from test_chat.py rather than recording new ones. Adds a tool.uv.sources override so the package installs the local (unpublished) semantic-conventions-ai source. This package never sets gen_ai.operation.name or gen_ai.provider.name on its spans, only the legacy gen_ai.system attribute, so both tests key spans on identified_by="gen_ai.system". gen_ai.response.finish_reasons is not emitted in either the legacy or streaming path -- a #4362-class gap, recorded but not fixed here.
Wires the conformance harness into cohere's test suite in warn-only mode (ENFORCING = False), reusing existing VCR cassettes from test_chat.py (v1 and v2 clients) rather than recording new ones. Adds a tool.uv.sources override so the package installs the local (unpublished) semantic-conventions-ai source. This package never sets gen_ai.operation.name or gen_ai.provider.name on its spans, only the legacy gen_ai.system attribute, so both tests key spans on identified_by="gen_ai.system". Unlike ollama, cohere DOES emit gen_ai.response.finish_reasons.
Wires the conformance harness into mistralai's test suite in warn-only mode
(ENFORCING = False), reusing existing VCR cassettes from test_chat.py rather
than recording new ones. Adds a tool.uv.sources override so the package
installs the local (unpublished) semantic-conventions-ai source.
This package never sets gen_ai.operation.name or gen_ai.provider.name on its
spans, only the legacy gen_ai.system attribute, so both tests key spans on
identified_by="gen_ai.system". gen_ai.response.finish_reasons is not emitted;
the finish reason only appears under the undeclared, indexed
gen_ai.completion.{n}.finish_reason -- a #4362-class gap.
Also fixes a latent bug in test_mistralai_chat_with_cache_tokens: it asserted
on SpanAttributes.GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS, which does not exist
on that class (the instrumentation actually sets
GenAIAttributes.GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS). This was masked before
because the package resolved a stale published semconv-ai wheel instead of
the local source; switching to the local source via tool.uv.sources exposed
it.
Wires the conformance harness into replicate's test suite in warn-only mode (ENFORCING = False), reusing an existing VCR cassette from test_llama.py rather than recording a new one. Adds a tool.uv.sources override so the package installs the local (unpublished) semantic-conventions-ai source. This package never sets gen_ai.operation.name or gen_ai.provider.name on its spans, only the legacy gen_ai.system attribute, so the test keys spans on identified_by="gen_ai.system". Of the whole contract this package only ever emits gen_ai.request.model on the span -- no response model/id, no token usage, and no gen_ai.response.finish_reasons.
Wires the conformance harness into alephalpha's test suite in warn-only mode (ENFORCING = False), reusing an existing VCR cassette from test_completion.py rather than recording a new one. Adds a tool.uv.sources override so the package installs the local (unpublished) semantic-conventions-ai source. This package never sets gen_ai.operation.name or gen_ai.provider.name on its spans, only the legacy gen_ai.system attribute, so the test keys spans on identified_by="gen_ai.system". gen_ai.response.finish_reasons is not emitted as a span attribute -- it only reaches the gen_ai.choice log event.
Wires the conformance harness into google-generativeai's test suite in warn-only mode (ENFORCING = False), reusing an existing VCR cassette from test_generate_content.py rather than recording a new one. Adds a tool.uv.sources override so the package installs the local (unpublished) semantic-conventions-ai source. Unlike the other five packages in this rollout, this instrumentation already targets the newer GenAI conventions directly, setting gen_ai.operation.name and gen_ai.provider.name on every span, so no identified_by override is needed. It also already emits gen_ai.response.finish_reasons, gen_ai.response.id, gen_ai.input.messages, and gen_ai.output.messages. The conformance test produces zero violations against this cassette. Also fixes a latent bug in two span_utils cache-token unit tests: they asserted on SpanAttributes.GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS, which does not exist on that class (the instrumentation actually sets GenAIAttributes.GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS). This was masked before because the package resolved a stale published semconv-ai wheel instead of the local source; switching to the local source via tool.uv.sources exposed it (same issue as the mistralai package in this rollout).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds a semconv-contract job that regenerates _contract/generated.py from the vendored .semconv/ registry via weaver and fails the build on any diff, closing the gap where the committed contract could drift from the registry or be hand-edited to weaken it undetected.
The step as written appends to the working tree, but make check regenerates before diffing, so an uncommitted tamper is always overwritten and the test passes vacuously. The real threat is a committed hand-edit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…extension limits Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reuses test_chat.py's cassettes; no finish_reasons gap for this package (emitted on both legacy and streaming paths). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reuses test_anthropic.py's cassettes; no finish_reasons gap for this package (emitted on both the legacy text_completion and Claude-3 chat/streaming paths). Also fixes test_prompt_caching.py::test_invoke_model_cache_tokens, broken by pinning to the local editable semconv-ai package (required so the conformance module resolves): the test referenced SpanAttributes.GEN_AI_USAGE_CACHE_*_TOKENS, removed from opentelemetry-semantic-conventions-ai in #4243 in favor of GenAIAttributes.GEN_AI_USAGE_CACHE_*_TOKENS (same wire value). The actual instrumentation (prompt_caching.py) already emits the correct attribute; only the test's import was stale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reuses test_chat_tracing.py's cassettes; no finish_reasons gap for this package (emitted on both legacy and streaming paths). Also fixes test_span_utils.py, broken by pinning to the local editable semconv-ai package (required so the conformance module resolves): several tests referenced SpanAttributes.GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS, removed from opentelemetry-semantic-conventions-ai in #4243 in favor of GenAIAttributes.GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS (same wire value). One of the six could not simply be corrected: set_model_response_attributes (the non-streaming path in span_utils.py) itself still references the removed SpanAttributes constant, so the AttributeError is silently swallowed by @dont_throw and the attribute never reaches the span — a genuine, pre-existing source bug outside the "tests only" scope of this batch. That one test is marked xfail(strict=True) with a comment explaining the gap rather than silently deleted or left crashing; the streaming counterpart already uses the correct constant and is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Batch 3: openai, bedrock, groq wired and tagged semconv:warn (16 packages total). watsonx skipped — its test dependency doesn't provide the module the instrumentation patches, so its whole suite already passes vacuously; fixing that needs a new dependency and is out of scope here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe change adds a pinned semantic-convention registry and generated Python contract. It adds conformance checking, CI freshness validation, rollout documentation, and warn-only tests across multiple GenAI instrumentation packages. ChangesSemantic-convention contract
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant InstrumentationTest
participant SharedConformanceHelper
participant ConformanceChecker
participant GeneratedContract
InstrumentationTest->>SharedConformanceHelper: collect exported spans
SharedConformanceHelper->>GeneratedContract: select contract group
SharedConformanceHelper->>ConformanceChecker: validate span attributes
ConformanceChecker-->>InstrumentationTest: violations or warnings
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 16
🧹 Nitpick comments (3)
docs/superpowers/plans/2026-08-02-semconv-contract.md (1)
1602-1605: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winPin the
uvversion used by CI.The job pins Weaver and registry references but installs
uvas"latest". A futureuvrelease can change resolver or environment behavior. Pin the repository’s tested version, or add a repository-level version pin.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/superpowers/plans/2026-08-02-semconv-contract.md` around lines 1602 - 1605, Update the Install uv step to use the repository’s tested, explicit uv version instead of "latest"; if no workflow-local version exists, reference the repository-level uv version pin consistently with the existing Weaver and registry pins..semconv/Makefile (1)
27-33: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winMake the
$(FILTERED)rule idempotent.
$(FILTERED)is a directory target. Make treats it as up to date whenever the directory exists. If a previous run fails aftermkdir -p $(dir $(FILTERED))or duringcp, the directory can remain in a partial state. The nextmake resolvethen skips the rule and resolves against incomplete upstream models. Two further hazards exist in the same rule:git clonefails when$(UPSTREAM)already exists and is non-empty, andcp -r $(UPSTREAM)/model $(FILTERED)nests amodel/directory inside$(FILTERED)when$(FILTERED)already exists.Remove both directories at the start of the rule.
♻️ Proposed fix
$(FILTERED): - mkdir -p $(BUILD) + rm -rf $(UPSTREAM) $(FILTERED) + mkdir -p $(BUILD) git clone -q --depth 1 --branch $(SEMCONV_VERSION) \ https://github.com/open-telemetry/semantic-conventions.git $(UPSTREAM) - mkdir -p $(dir $(FILTERED)) cp -r $(UPSTREAM)/model $(FILTERED) cd $(FILTERED) && rm -rf gen-ai mcp openai🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.semconv/Makefile around lines 27 - 33, Make the $(FILTERED) rule idempotent by removing both $(UPSTREAM) and $(FILTERED) at the start of the recipe, before creating directories or cloning. Preserve the existing clone, copy, and filtering steps so each run starts from clean directories and copies the upstream model directly into a fresh $(FILTERED) target.packages/opentelemetry-instrumentation-groq/tests/traces/test_span_utils.py (1)
476-493: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winConfirm intent to defer the underlying cache-token bug.
The
xfail(strict=True)correctly documents thatset_model_response_attributes(non-streaming path) still uses the removedSpanAttributes.GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS, sogen_ai.usage.cache_read_input_tokensis silently dropped for non-streaming Groq responses with cached prompt tokens. The streaming path already usesGenAIAttributescorrectly.This is a real, silent telemetry data-loss bug in production code, even though it is pre-existing and out of scope for this test-only PR. Since the fix looks like a one-line constant swap (mirroring the streaming path), do you want me to generate that fix for
span_utils.py, or open a tracking issue for it?🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/opentelemetry-instrumentation-groq/tests/traces/test_span_utils.py` around lines 476 - 493, Confirm the test-only scope by retaining the strict xfail and its documented reason; do not modify production code in this change. Track the underlying non-streaming cache-token issue separately, specifically the stale constant used by set_model_response_attributes, while preserving the existing streaming implementation.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yml:
- Around line 199-206: Update the semconv-contract job to declare explicit
read-only repository permissions matching its needs, and configure its
actions/checkout@v4 step with credential persistence disabled. Keep the existing
checkout ref and fetch-depth settings unchanged.
In @.semconv/Makefile:
- Around line 47-50: Update the .semconv validation flow around the check target
so it also verifies that registry/ matches the pinned SEMCONV_GENAI_REF, not
only that generated.py matches the registry. Add or invoke a verify-vendor
target that clones the pinned revision into a temporary directory and diffs it
against the committed registry, while preserving the existing generated-artifact
check.
In @.semconv/README.md:
- Around line 3-5: Update the README description near the vendor SHA reference
to limit the conformance claim to instrumentation packages that have conformance
coverage, rather than stating that every package is tested against the contract.
In `@docs/superpowers/plans/2026-08-02-semconv-contract.md`:
- Around line 1631-1638: Update the tamper-test procedure in the contract
documentation to require a clean repository, execute all edits and validation
inside a temporary worktree, and register trap-based cleanup so the worktree is
removed even when a command fails. Replace the destructive git reset flow while
preserving the generated contract tampering and make -C .semconv check
verification.
- Around line 1631-1638: Update the tamper-test command sequence around the
`make -C .semconv check` invocation to capture its non-zero status immediately,
restore the worktree with `git reset --hard HEAD~1`, and then exit using the
saved status instead of allowing `echo` or cleanup commands to replace it.
- Around line 1231-1233: Update the pytest command examples around the
conformance test invocation to preserve pytest’s exit status through the tail
pipeline by enabling pipefail or explicitly capturing and returning pytest’s
status. Apply the same correction to the additional command range, while
retaining the truncated output and warn-only behavior.
- Around line 1300-1308: Update the shell commands around the package-scope
checks to avoid the hardcoded /Users/gal.kleinman/dev/openllmetry path. Run them
from the repository root or derive and use the root via git rev-parse
--show-toplevel, including the related commands near the additional referenced
section.
- Around line 1252-1256: Update the Step 5 instruction to use the
repository-approved uv run form for regenerating the lockfile instead of
invoking uv lock directly. Keep the dependency verification and expected-result
guidance unchanged, and do not add an exception unless the command cannot be
expressed through uv run.
- Around line 734-738: Update the Task 4 interface contract to include the
implemented assert_conforms expected parameter and replace the Violation kind
bad_enum_value with missing_expected and unknown_enum_value. Keep the listed
signatures and downstream references consistent with the amended conformance
API.
- Around line 1588-1596: Update the semconv-contract job’s checkout
configuration to avoid relying on the unavailable pull_request head SHA during
push events: either restrict the job to pull_request events or change the ref
expression to fall back to github.sha. Preserve checkout of the PR head when
running for pull requests.
- Around line 1243-1245: Update the documented sed commands around the
conformance test instructions to use a portable in-place editing approach, or
explicitly provide separate macOS/BSD and GNU sed forms. Ensure both ENFORCING
toggles are covered without relying on BSD-only `sed -i ''` syntax.
- Around line 1440-1441: Use project.json semconv tags as the sole source of
enforcement mode instead of maintaining the duplicate ENFORCING boolean. Update
docs/superpowers/plans/2026-08-02-semconv-contract.md at lines 1440-1441 and
1539-1544, and docs/superpowers/specs/2026-07-27-ai-native-maintenance-design.md
at lines 142-146, removing or replacing package-level ENFORCING references with
tag-driven behavior so enforcing tags cannot run in warn-only mode.
- Around line 1102-1116: Expand the telemetry extension contract around
gen_ai.headers, gen_ai.user, and gen_ai.completion with explicit opt-in
allowlisting, header redaction for authorization, cookies, and API keys,
content-capture controls, and retention/export restrictions. Add tests covering
omission or rejection of those sensitive headers and unapproved content while
preserving approved telemetry behavior.
In `@docs/superpowers/specs/2026-07-27-ai-native-maintenance-design.md`:
- Around line 62-65: Update the package layout reference in the design to use
the actual generated contract artifact path,
opentelemetry/semconv_ai/_contract/generated.py, matching the generator output
and import locations instead of the nonexistent .../_generated/contract.py path.
- Around line 121-127: Resolve the payload-schema scope consistently across both
documents: in docs/superpowers/specs/2026-07-27-ai-native-maintenance-design.md
lines 121-127, either remove the promise that the harness validates payloads
against upstream JSON Schemas or add the corresponding implementation and
acceptance tests; in docs/superpowers/plans/2026-08-02-semconv-contract.md lines
1683-1685, update the self-review and deferred-scope statement to match that
same choice.
In
`@packages/opentelemetry-semantic-conventions-ai/opentelemetry/semconv_ai/conformance.py`:
- Around line 135-174: Update assert_conforms so non-blocking violations are
reported with warnings.warn before enforcing mode raises for blocking
violations. Preserve the existing AssertionError for blocking findings, ensure
only non-blocking findings are warned in enforcing mode, and avoid warning when
no reportable violations exist.
---
Nitpick comments:
In @.semconv/Makefile:
- Around line 27-33: Make the $(FILTERED) rule idempotent by removing both
$(UPSTREAM) and $(FILTERED) at the start of the recipe, before creating
directories or cloning. Preserve the existing clone, copy, and filtering steps
so each run starts from clean directories and copies the upstream model directly
into a fresh $(FILTERED) target.
In `@docs/superpowers/plans/2026-08-02-semconv-contract.md`:
- Around line 1602-1605: Update the Install uv step to use the repository’s
tested, explicit uv version instead of "latest"; if no workflow-local version
exists, reference the repository-level uv version pin consistently with the
existing Weaver and registry pins.
In `@packages/opentelemetry-instrumentation-groq/tests/traces/test_span_utils.py`:
- Around line 476-493: Confirm the test-only scope by retaining the strict xfail
and its documented reason; do not modify production code in this change. Track
the underlying non-streaming cache-token issue separately, specifically the
stale constant used by set_model_response_attributes, while preserving the
existing streaming implementation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 89e0cb8c-32f3-4ee6-a842-a23c72c2c68b
⛔ Files ignored due to path filters (16)
packages/opentelemetry-instrumentation-alephalpha/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-anthropic/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-bedrock/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-cohere/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-google-generativeai/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-groq/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-langchain/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-llamaindex/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-mistralai/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-ollama/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-openai-agents/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-openai/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-replicate/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-sagemaker/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-together/uv.lockis excluded by!**/*.lockpackages/opentelemetry-instrumentation-writer/uv.lockis excluded by!**/*.lock
📒 Files selected for processing (90)
.github/workflows/ci.yml.gitignore.semconv/Makefile.semconv/README.md.semconv/registry/aws-bedrock/registry.yaml.semconv/registry/gen-ai/events.yaml.semconv/registry/gen-ai/gen-ai-input-messages.json.semconv/registry/gen-ai/gen-ai-memory-records.json.semconv/registry/gen-ai/gen-ai-output-messages.json.semconv/registry/gen-ai/gen-ai-retrieval-documents.json.semconv/registry/gen-ai/gen-ai-system-instructions.json.semconv/registry/gen-ai/gen-ai-tool-call-arguments.json.semconv/registry/gen-ai/gen-ai-tool-call-result.json.semconv/registry/gen-ai/gen-ai-tool-definitions.json.semconv/registry/gen-ai/metrics.yaml.semconv/registry/gen-ai/registry.yaml.semconv/registry/gen-ai/spans.yaml.semconv/registry/manifest.yaml.semconv/registry/mcp/common.yaml.semconv/registry/mcp/metrics.yaml.semconv/registry/mcp/registry.yaml.semconv/registry/mcp/spans.yaml.semconv/registry/openai/registry.yaml.semconv/versions.envdocs/ai/semconv-rollout.mddocs/superpowers/plans/2026-08-02-semconv-contract.mddocs/superpowers/specs/2026-07-27-ai-native-maintenance-design.mdpackages/opentelemetry-instrumentation-alephalpha/project.jsonpackages/opentelemetry-instrumentation-alephalpha/pyproject.tomlpackages/opentelemetry-instrumentation-alephalpha/tests/test_conformance.pypackages/opentelemetry-instrumentation-anthropic/project.jsonpackages/opentelemetry-instrumentation-anthropic/pyproject.tomlpackages/opentelemetry-instrumentation-anthropic/tests/test_conformance.pypackages/opentelemetry-instrumentation-bedrock/project.jsonpackages/opentelemetry-instrumentation-bedrock/pyproject.tomlpackages/opentelemetry-instrumentation-bedrock/tests/traces/test_conformance.pypackages/opentelemetry-instrumentation-bedrock/tests/traces/test_prompt_caching.pypackages/opentelemetry-instrumentation-cohere/project.jsonpackages/opentelemetry-instrumentation-cohere/pyproject.tomlpackages/opentelemetry-instrumentation-cohere/tests/test_conformance.pypackages/opentelemetry-instrumentation-google-generativeai/project.jsonpackages/opentelemetry-instrumentation-google-generativeai/pyproject.tomlpackages/opentelemetry-instrumentation-google-generativeai/tests/test_conformance.pypackages/opentelemetry-instrumentation-google-generativeai/tests/test_generate_content.pypackages/opentelemetry-instrumentation-groq/project.jsonpackages/opentelemetry-instrumentation-groq/pyproject.tomlpackages/opentelemetry-instrumentation-groq/tests/traces/test_conformance.pypackages/opentelemetry-instrumentation-groq/tests/traces/test_span_utils.pypackages/opentelemetry-instrumentation-langchain/project.jsonpackages/opentelemetry-instrumentation-langchain/pyproject.tomlpackages/opentelemetry-instrumentation-langchain/tests/test_conformance.pypackages/opentelemetry-instrumentation-llamaindex/project.jsonpackages/opentelemetry-instrumentation-llamaindex/pyproject.tomlpackages/opentelemetry-instrumentation-llamaindex/tests/test_conformance.pypackages/opentelemetry-instrumentation-mistralai/project.jsonpackages/opentelemetry-instrumentation-mistralai/pyproject.tomlpackages/opentelemetry-instrumentation-mistralai/tests/test_chat.pypackages/opentelemetry-instrumentation-mistralai/tests/test_conformance.pypackages/opentelemetry-instrumentation-ollama/project.jsonpackages/opentelemetry-instrumentation-ollama/pyproject.tomlpackages/opentelemetry-instrumentation-ollama/tests/test_conformance.pypackages/opentelemetry-instrumentation-openai-agents/project.jsonpackages/opentelemetry-instrumentation-openai-agents/pyproject.tomlpackages/opentelemetry-instrumentation-openai-agents/tests/test_conformance.pypackages/opentelemetry-instrumentation-openai/project.jsonpackages/opentelemetry-instrumentation-openai/pyproject.tomlpackages/opentelemetry-instrumentation-openai/tests/traces/test_conformance.pypackages/opentelemetry-instrumentation-replicate/project.jsonpackages/opentelemetry-instrumentation-replicate/pyproject.tomlpackages/opentelemetry-instrumentation-replicate/tests/test_conformance.pypackages/opentelemetry-instrumentation-sagemaker/project.jsonpackages/opentelemetry-instrumentation-sagemaker/pyproject.tomlpackages/opentelemetry-instrumentation-sagemaker/tests/test_conformance.pypackages/opentelemetry-instrumentation-together/project.jsonpackages/opentelemetry-instrumentation-together/pyproject.tomlpackages/opentelemetry-instrumentation-together/tests/test_conformance.pypackages/opentelemetry-instrumentation-writer/project.jsonpackages/opentelemetry-instrumentation-writer/pyproject.tomlpackages/opentelemetry-instrumentation-writer/tests/test_conformance.pypackages/opentelemetry-semantic-conventions-ai/opentelemetry/semconv_ai/_contract/__init__.pypackages/opentelemetry-semantic-conventions-ai/opentelemetry/semconv_ai/_contract/_generator.pypackages/opentelemetry-semantic-conventions-ai/opentelemetry/semconv_ai/_contract/extensions.pypackages/opentelemetry-semantic-conventions-ai/opentelemetry/semconv_ai/_contract/generated.pypackages/opentelemetry-semantic-conventions-ai/opentelemetry/semconv_ai/_testing_conformance.pypackages/opentelemetry-semantic-conventions-ai/opentelemetry/semconv_ai/conformance.pypackages/opentelemetry-semantic-conventions-ai/pyproject.tomlpackages/opentelemetry-semantic-conventions-ai/tests/test_conformance.pypackages/opentelemetry-semantic-conventions-ai/tests/test_contract_model.pypackages/opentelemetry-semantic-conventions-ai/tests/test_extensions.pypackages/opentelemetry-semantic-conventions-ai/tests/test_generator.py
Comment on lines
+1102
to
+1116
| "gen_ai.user": ( | ||
| "End-user identifier passed through from provider SDKs. No upstream " | ||
| "equivalent in the GenAI registry at the pinned ref." | ||
| ), | ||
| "gen_ai.headers": ( | ||
| "Request headers captured for debugging. Opt-in only. No upstream equivalent." | ||
| ), | ||
| "gen_ai.is_streaming": ( | ||
| "Whether the request used streaming. Upstream expresses this through span " | ||
| "structure rather than an attribute." | ||
| ), | ||
| "gen_ai.completion": ( | ||
| "Legacy prompt/completion content attribute, predating upstream's " | ||
| "gen_ai.output.messages. Retained for backwards compatibility; migration " | ||
| "to gen_ai.output.messages is the intended path." |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== file existence and line count =="
if [ -f docs/superpowers/plans/2026-08-02-semconv-contract.md ]; then
wc -l docs/superpowers/plans/2026-08-02-semconv-contract.md
sed -n '1050,1140p' docs/superpowers/plans/2026-08-02-semconv-contract.md | cat -n
else
echo "missing docs/superpowers/plans/2026-08-02-semconv-contract.md"
git ls-files | rg 'semconv|gen_ai|telemetry|header|redact' || true
fi
echo "== search gen_ai attributes and redaction mentions =="
rg -n --hidden --glob '!vendor/**' --glob '!node_modules/**' \
'gen_ai\.(user|headers|completion|input\.messages|output\.messages)|redact|retention|authorization|cookie|api[_-]?key|Authorization|Cookie|api[_-]?key|PII|personal|sensitive|header' . || true
echo "== relevant docs files =="
git ls-files | rg '(^docs/|docs/.*)semconv|telemetry|observability|gen_ai|llm|agent' || trueRepository: traceloop/openllmetry
Length of output: 50378
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== target sections =="
sed -n '1070,1125p' docs/superpowers/plans/2026-08-02-semconv-contract.md | cat -n
echo "== semconv rollout related target sections =="
sed -n '1,220p' docs/ai/semconv-rollout.md | cat -n
echo "== sensitive header references excluding yaml cassettes/base64-like output =="
rg -n --max-count 200 \
'gen_ai\.headers|filter_headers|authorization|Authorization|cookie|Cookie|api[_-]?key|Api-Key|Set-Cookie|redact|sensitive|pii|PII|keep_content|content_capture|prompt|completion|gen_ai\.user|gen_ai\.completion' \
. \
--glob '!**/*.yaml' --glob '!**/*.yml' --glob '!**/cassette*' --glob '!**/docs/**' || true
echo "== files defining gen_ai.user/message attributes =="
rg -n --max-count 200 'gen_ai\.user|gen_ai\.user\.message|gen_ai\.messages|gen_ai\.input\.messages|gen_ai\.output\.messages' packages docs --glob '!**/cassettes/**' --glob '!**/*.yaml' || trueRepository: traceloop/openllmetry
Length of output: 50378
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== target document sections =="
sed -n '1050,1135p' docs/superpowers/plans/2026-08-02-semconv-contract.md | cat -n
echo "== docs/ai/semconv-rollout.md summary =="
sed -n '1,220p' docs/ai/semconv-rollout.md | cat -n
echo "== implementation references excluding cassettes/yamls"
rg -n --max-count 80 \
'gen_ai\.headers|gen_ai\.user|gen_ai\.completion|filter_headers|redact|keep_content|capture_headers|allowed_headers|authorization|Authorization|cookie|Cookie|api[_-]?key' \
packages --glob '!**/cassettes/**' --glob '!**/*.yaml' --glob '!**/*.yml' \
--glob '!**/data/**' --glob '!**/tests/**/cassettes/**' || true
echo "== documentation references excluding cassettes/yamls/data"
rg -n --max-count 80 \
'gen_ai\.headers|gen_ai\.user|gen_ai\.completion|filter_headers|redact|keep_content|capture_headers|allowed_headers|authorization|Authorization|pii|PII|sensitive|cookie|Cookie|api[_-]?key' \
docs --glob '!**/cassettes/**' --glob '!**/*.yaml' --glob '!**/*.yml' --glob '!**/data/**' || true
echo "== tests matching sensitive header/content redaction excluding cassettes =="
rg -n --max-count 80 \
'filter_headers|authorization|Authorization|set-cookie|Cookie|api[_-]?key|redact|keep_content|capture_header|capture_headers' \
packages --glob '*.py' --glob '!**/cassettes/**' --glob '!**/*.yaml' --glob '!**/*.yml' || trueRepository: traceloop/openllmetry
Length of output: 50380
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== semantic conventions definitions =="
sed -n '55,175p' packages/opentelemetry-semantic-conventions-ai/opentelemetry/semconv_ai/__init__.py | cat -n
echo "== core utility/header filtering references excluding tests/cassettes/yaml/data =="
rg -n --max-count 120 \
'capture_headers|allowed_headers|filter_headers|redactions|REDACTIONS|redact|API_KEY|api[_-]?key|authorization|Authorization|cookie|Cookie|set-cookie|Set-Cookie|gen_ai\.headers|gen_ai\.user|gen_ai\.completion|keep_content' \
packages/opentelemetry-instrumentation-shared packages/opentelemetry-semantic-conventions-ai \
--glob '*.py' --glob '!**/tests/**' --glob '!**/cassettes/**' --glob '!**/*.yaml' --glob '!**/*.yml' --glob '!**/data/**' || true
echo "== focused shared instrumentation config/utility files =="
git ls-files packages/opentelemetry-instrumentation-shared | sed -n '1,120p'
echo "== emit/filter related tests for headers/content redaction =="
rg -n --max-count 120 \
'capture_headers|allowed_headers|filter_headers|redact|REDACTIONS|gen_ai\.headers|gen_ai\.user|content|keep_content' \
packages/opentelemetry-instrumentation-shared tests packages --glob '*.py' --glob '!**/cassettes/**' --glob '!**/*.yaml' --glob '!**/*.yml' --glob '!**/data/**' || trueRepository: traceloop/openllmetry
Length of output: 50378
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== semantic conventions constants =="
sed -n '65,175p' packages/opentelemetry-semantic-conventions-ai/opentelemetry/semconv_ai/__init__.py | cat -n
echo "== shared instrumentation files =="
git ls-files 'packages/opentelemetry-instrumentation-shared/opentelemetry/instrumentation/shared/**' | sed -n '1,160p'
echo "== search shared package for header/content/redaction/api references =="
rg -n --max-count 120 \
'capture_headers|allowed_headers|filter_headers|allowed_headers|redact|REDACTIONS|api[_-]?key|API_KEY|api_key|authorization|Authorization|cookie|Cookie|set-cookie|Set-Cookie|keep_content|content_capture|prompt|completion|filter_post_data_parameters|filter_query_parameters' \
packages/opentelemetry-instrumentation-shared packages/opentelemetry-instrumentation-base \
--glob '*.py' --glob '!**/tests/**' --glob '!**/cassettes/**' --glob '!**/*.yaml' --glob '!**/*.yml' || true
echo "== focused tests for config handling and redaction =="
rg -n --max-count 120 \
'capture_headers|allowed_headers|filter_headers|filter_post_data_parameters|filter_query_parameters|redact|api[_-]?key|authorization|cookie|set-cookie|keep_content|enable_content_tracing' \
packages/opentelemetry-instrumentation-shared tests --glob '*.py' --glob '!**/cassettes/**' --glob '!**/*.yaml' --glob '!**/*.yml' || trueRepository: traceloop/openllmetry
Length of output: 6988
Add redaction and retention rules for the opt-in telemetry extensions.
gen_ai.headers, gen_ai.user, and gen_ai.completion can carry PII, credentials, session cookies, or raw prompt content. The contract should expand the opt-in rationale into an explicit allowlist, header redaction rules, content-capture controls, and retention/export rules; add tests that reject/omit authorization headers, cookies, API keys, and unapproved content.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/superpowers/plans/2026-08-02-semconv-contract.md` around lines 1102 -
1116, Expand the telemetry extension contract around gen_ai.headers,
gen_ai.user, and gen_ai.completion with explicit opt-in allowlisting, header
redaction for authorization, cookies, and API keys, content-capture controls,
and retention/export restrictions. Add tests covering omission or rejection of
those sensitive headers and unapproved content while preserving approved
telemetry behavior.
Comment on lines
+1588
to
+1596
| semconv-contract: | ||
| name: Semconv Contract | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
| ref: ${{ github.event.pull_request.head.sha }} | ||
|
|
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 4 'pull_request|push|workflow_dispatch|github\.event\.pull_request\.head\.sha' .github/workflows/ci.ymlRepository: traceloop/openllmetry
Length of output: 1411
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workflow workflow names and event refs =="
rg -n -C 8 '^name:|^on:|pull_request|push|workflow_dispatch|github\.event\.pull_request\.head\.sha|ref: \$\{\{ github\.event\.pull_request\.head\.sha \}\}' .github/workflows/ci.yml
echo
echo "== all workflow files in repository =="
git ls-files .github/workflows || true
echo
echo "== docs file context around reported lines =="
num_lines=$(wc -l < docs/superpowers/plans/2026-08-02-semconv-contract.md)
start=$((1588))
end=$((1605))
sed -n "${start},${end}p" docs/superpowers/plans/2026-08-02-semconv-contract.md | cat -nRepository: traceloop/openllmetry
Length of output: 3175
Restrict this job to pull requests when referencing github.event.pull_request.head.sha.
.github/workflows/ci.yml launches the same semconv-contract job from both pull_request and push events on main. On push, this expression has no value, so move the job into a PR-only conditional or use an event-safe checkout SHA, such as github.event.pull_request.head.sha || github.sha.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/superpowers/plans/2026-08-02-semconv-contract.md` around lines 1588 -
1596, Update the semconv-contract job’s checkout configuration to avoid relying
on the unavailable pull_request head SHA during push events: either restrict the
job to pull_request events or change the ref expression to fall back to
github.sha. Preserve checkout of the PR head when running for pull requests.
Comment on lines
+1631
to
+1638
| ```bash | ||
| cd /Users/gal.kleinman/dev/openllmetry | ||
| GEN=packages/opentelemetry-semantic-conventions-ai/opentelemetry/semconv_ai/_contract/generated.py | ||
| # weaken the contract the way a bad hand-edit would | ||
| sed -i '' "0,/Level.REQUIRED/s//Level.OPT_IN/" "$GEN" | ||
| git add "$GEN" && git commit -q -m "TEMP tamper test" | ||
| make -C .semconv check; echo "EXIT=$?" | ||
| git reset --hard HEAD~1 |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Do not discard unrelated local changes.
git reset --hard HEAD~1 can delete pre-existing work and does not restore state if an earlier command fails. Require a clean tree and run the tamper test in a temporary worktree with cleanup handled by a trap.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/superpowers/plans/2026-08-02-semconv-contract.md` around lines 1631 -
1638, Update the tamper-test procedure in the contract documentation to require
a clean repository, execute all edits and validation inside a temporary
worktree, and register trap-based cleanup so the worktree is removed even when a
command fails. Replace the destructive git reset flow while preserving the
generated contract tampering and make -C .semconv check verification.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Preserve the non-zero result from make check.
make -C .semconv check; echo "EXIT=$?" prints the status, but echo returns zero. The following command can also replace the exit status. Save the status, restore the worktree, then execute exit "$status".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/superpowers/plans/2026-08-02-semconv-contract.md` around lines 1631 -
1638, Update the tamper-test command sequence around the `make -C .semconv
check` invocation to capture its non-zero status immediately, restore the
worktree with `git reset --hard HEAD~1`, and then exit using the saved status
instead of allowing `echo` or cleanup commands to replace it.
Comment on lines
+62
to
+65
| packages/opentelemetry-semantic-conventions-ai/ | ||
| .../_generated/contract.py weaver-generated: required/recommended attrs per | ||
| span kind, enum values, payload JSON Schemas | ||
| .../conformance.py hand-written harness consuming contract.py |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Use the actual contract artifact path.
The design names .../_generated/contract.py, but the implementation plan creates opentelemetry/semconv_ai/_contract/generated.py. The current path points agents to a nonexistent module. Replace it with the exact repository path used by the generator and imports.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/superpowers/specs/2026-07-27-ai-native-maintenance-design.md` around
lines 62 - 65, Update the package layout reference in the design to use the
actual generated contract artifact path,
opentelemetry/semconv_ai/_contract/generated.py, matching the generator output
and import locations instead of the nonexistent .../_generated/contract.py path.
Comment on lines
+121
to
+127
| `_generated/contract.py` is **committed to the repo**, not generated at test time. CI | ||
| regenerates and fails on diff. This makes every contract change a reviewable line-level | ||
| diff rather than an invisible behavioural shift, and keeps test runs offline. | ||
|
|
||
| Each instrumentation package gets one test importing the shared harness, parametrized over | ||
| the spans it emits. The harness validates emitted attributes against the generated | ||
| contract and validates message payloads against upstream's JSON Schemas directly. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Resolve the payload-schema scope before merge.
The design says the harness validates message payloads against upstream JSON Schemas. The plan explicitly defers that validation. Choose one contract and update the implementation plan and acceptance tests.
docs/superpowers/specs/2026-07-27-ai-native-maintenance-design.md#L121-L127: remove JSON-Schema validation from the contract promise, or add it to the implementation and tests.docs/superpowers/plans/2026-08-02-semconv-contract.md#L1683-L1685: align the self-review and deferred-scope statement with the chosen behavior.
🧰 Tools
🪛 LanguageTool
[grammar] ~125-~125: Ensure spelling is correct
Context: ... one test importing the shared harness, parametrized over the spans it emits. The harness va...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
📍 Affects 2 files
docs/superpowers/specs/2026-07-27-ai-native-maintenance-design.md#L121-L127(this comment)docs/superpowers/plans/2026-08-02-semconv-contract.md#L1683-L1685
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/superpowers/specs/2026-07-27-ai-native-maintenance-design.md` around
lines 121 - 127, Resolve the payload-schema scope consistently across both
documents: in docs/superpowers/specs/2026-07-27-ai-native-maintenance-design.md
lines 121-127, either remove the promise that the harness validates payloads
against upstream JSON Schemas or add the corresponding implementation and
acceptance tests; in docs/superpowers/plans/2026-08-02-semconv-contract.md lines
1683-1685, update the self-review and deferred-scope statement to match that
same choice.
Comment on lines
+135
to
+174
| def assert_conforms( | ||
| span: Any, | ||
| group_id: str, | ||
| *, | ||
| enforcing: bool, | ||
| extensions: FrozenSet[str] = frozenset(), | ||
| expected: FrozenSet[str] = frozenset(), | ||
| _spans: Optional[Dict[str, SpanSpec]] = None, | ||
| ) -> List[Violation]: | ||
| """Check one span against a contract group. | ||
|
|
||
| Enforcing mode raises on blocking violations. Non-blocking findings always warn, | ||
| never fail, in either mode. Returns every violation found. | ||
| """ | ||
| table = SPANS if _spans is None else _spans | ||
| if group_id not in table: | ||
| raise KeyError(f"no such span group in the contract: {group_id!r}") | ||
|
|
||
| violations = check_attributes( | ||
| dict(span.attributes or {}), table[group_id], extensions, expected | ||
| ) | ||
| if not violations: | ||
| return [] | ||
|
|
||
| span_name = getattr(span, "name", "<unnamed>") | ||
| blocking = [v for v in violations if v.blocking] | ||
|
|
||
| if enforcing and blocking: | ||
| report = "\n".join(f" {v}" for v in blocking) | ||
| raise AssertionError(f"span {span_name!r} violates {group_id}:\n{report}") | ||
|
|
||
| reportable = violations if not enforcing else [v for v in violations if not v.blocking] | ||
| if reportable: | ||
| report = "\n".join(f" {v}" for v in reportable) | ||
| warnings.warn( | ||
| f"span {span_name!r} violates {group_id}:\n{report}", | ||
| ConformanceWarning, | ||
| stacklevel=2, | ||
| ) | ||
| return violations |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Enforcing mode drops non-blocking warnings when blocking violations exist.
The docstring states non-blocking findings "always warn, never fail, in either mode." Line 162 raises AssertionError immediately when enforcing and blocking, before the code ever reaches the warnings.warn call at line 169. As a result, when a span has both a blocking violation (e.g. MISSING_REQUIRED) and a non-blocking one (e.g. UNKNOWN_ENUM_VALUE), only the blocking violation is surfaced in the raised exception; the non-blocking finding is silently dropped instead of warning as documented.
Warn about the non-blocking violations before raising, so enforcing mode matches its documented contract.
🐛 Proposed fix to warn on non-blocking violations before raising
if enforcing and blocking:
+ non_blocking = [v for v in violations if not v.blocking]
+ if non_blocking:
+ report = "\n".join(f" {v}" for v in non_blocking)
+ warnings.warn(
+ f"span {span_name!r} violates {group_id}:\n{report}",
+ ConformanceWarning,
+ stacklevel=2,
+ )
report = "\n".join(f" {v}" for v in blocking)
raise AssertionError(f"span {span_name!r} violates {group_id}:\n{report}")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def assert_conforms( | |
| span: Any, | |
| group_id: str, | |
| *, | |
| enforcing: bool, | |
| extensions: FrozenSet[str] = frozenset(), | |
| expected: FrozenSet[str] = frozenset(), | |
| _spans: Optional[Dict[str, SpanSpec]] = None, | |
| ) -> List[Violation]: | |
| """Check one span against a contract group. | |
| Enforcing mode raises on blocking violations. Non-blocking findings always warn, | |
| never fail, in either mode. Returns every violation found. | |
| """ | |
| table = SPANS if _spans is None else _spans | |
| if group_id not in table: | |
| raise KeyError(f"no such span group in the contract: {group_id!r}") | |
| violations = check_attributes( | |
| dict(span.attributes or {}), table[group_id], extensions, expected | |
| ) | |
| if not violations: | |
| return [] | |
| span_name = getattr(span, "name", "<unnamed>") | |
| blocking = [v for v in violations if v.blocking] | |
| if enforcing and blocking: | |
| report = "\n".join(f" {v}" for v in blocking) | |
| raise AssertionError(f"span {span_name!r} violates {group_id}:\n{report}") | |
| reportable = violations if not enforcing else [v for v in violations if not v.blocking] | |
| if reportable: | |
| report = "\n".join(f" {v}" for v in reportable) | |
| warnings.warn( | |
| f"span {span_name!r} violates {group_id}:\n{report}", | |
| ConformanceWarning, | |
| stacklevel=2, | |
| ) | |
| return violations | |
| def assert_conforms( | |
| span: Any, | |
| group_id: str, | |
| *, | |
| enforcing: bool, | |
| extensions: FrozenSet[str] = frozenset(), | |
| expected: FrozenSet[str] = frozenset(), | |
| _spans: Optional[Dict[str, SpanSpec]] = None, | |
| ) -> List[Violation]: | |
| """Check one span against a contract group. | |
| Enforcing mode raises on blocking violations. Non-blocking findings always warn, | |
| never fail, in either mode. Returns every violation found. | |
| """ | |
| table = SPANS if _spans is None else _spans | |
| if group_id not in table: | |
| raise KeyError(f"no such span group in the contract: {group_id!r}") | |
| violations = check_attributes( | |
| dict(span.attributes or {}), table[group_id], extensions, expected | |
| ) | |
| if not violations: | |
| return [] | |
| span_name = getattr(span, "name", "<unnamed>") | |
| blocking = [v for v in violations if v.blocking] | |
| if enforcing and blocking: | |
| non_blocking = [v for v in violations if not v.blocking] | |
| if non_blocking: | |
| report = "\n".join(f" {v}" for v in non_blocking) | |
| warnings.warn( | |
| f"span {span_name!r} violates {group_id}:\n{report}", | |
| ConformanceWarning, | |
| stacklevel=2, | |
| ) | |
| report = "\n".join(f" {v}" for v in blocking) | |
| raise AssertionError(f"span {span_name!r} violates {group_id}:\n{report}") | |
| reportable = violations if not enforcing else [v for v in violations if not v.blocking] | |
| if reportable: | |
| report = "\n".join(f" {v}" for v in reportable) | |
| warnings.warn( | |
| f"span {span_name!r} violates {group_id}:\n{report}", | |
| ConformanceWarning, | |
| stacklevel=2, | |
| ) | |
| return violations |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@packages/opentelemetry-semantic-conventions-ai/opentelemetry/semconv_ai/conformance.py`
around lines 135 - 174, Update assert_conforms so non-blocking violations are
reported with warnings.warn before enforcing mode raises for blocking
violations. Preserve the existing AssertionError for blocking findings, ensure
only non-blocking findings are warned in enforcing mode, and avoid warning when
no reportable violations exist.
Adds `make -C .semconv verify-vendor`, which re-clones the pinned upstream SHA into a scratch dir and diffs it against the committed registry/ tree. `check` alone only proves generated.py matches registry/ -- it never proves registry/ itself matches the pin, so a hand-edited registry plus a regenerated artifact previously passed `check` cleanly while silently weakening the contract. Wires the new target into the semconv-contract CI job ahead of the existing freshness check. Also gives semconv-contract an explicit `permissions: contents: read` block and `persist-credentials: false` on checkout, matching test-packages and addressing CodeQL alert 20 -- the job only reads the repo and has no reason to keep a persisted token around for its git clone / docker run steps. Corrects two inaccuracies in the semconv-contract plan doc: Task 4's Interfaces block described the pre-redesign conformance API (missing `expected`, a `bad_enum_value` kind that no longer exists); Task 8's tamper test used `git reset --hard` without a clean-tree guard, which would silently discard uncommitted work. Also narrows the .semconv/README claim that every instrumentation package is tested against the contract to packages with conformance coverage, pointing at docs/ai/semconv-rollout.md for current state.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/superpowers/plans/2026-08-02-semconv-contract.md`:
- Around line 736-738: Update the stale self-review statement near the
Violation-kind discussion to say there are four strings, matching the four
values defined by the conformance contract and preserving the existing kind
names and blocking behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 7505f1c2-f88e-41e3-bcba-eea8f3fa2ec8
📒 Files selected for processing (4)
.github/workflows/ci.yml.semconv/Makefile.semconv/README.mddocs/superpowers/plans/2026-08-02-semconv-contract.md
🚧 Files skipped from review as they are similar to previous changes (2)
- .github/workflows/ci.yml
- .semconv/README.md
Comment on lines
+736
to
+738
| - Produces: `Violation(kind, attribute, detail)`; `check_attributes(attributes, spec, extensions=frozenset(), expected=frozenset()) -> list[Violation]`; `ConformanceWarning`; `assert_conforms(span, group_id, *, enforcing, extensions=frozenset(), expected=frozenset(), _spans=None) -> list[Violation]`. Tasks 5–7 consume `assert_conforms`, `extensions`, and `expected`. | ||
|
|
||
| Violation `kind` is one of exactly four strings: `"missing_required"`, `"missing_expected"`, `"undeclared_gen_ai"` (all blocking), and `"unknown_enum_value"` (non-blocking). `BLOCKING_KINDS` excludes `"unknown_enum_value"`. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Update the stale Violation-kind count.
Line 738 defines four Violation.kind values, but Line 1694 still states that there are three strings. Change the self-review statement to four strings so the plan remains consistent with the conformance contract.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/superpowers/plans/2026-08-02-semconv-contract.md` around lines 736 -
738, Update the stale self-review statement near the Violation-kind discussion
to say there are four strings, matching the four values defined by the
conformance contract and preserving the existing kind names and blocking
behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements spec 1 of the AI-native maintenance design (
docs/superpowers/specs/2026-07-27-ai-native-maintenance-design.md).Turns the official OpenTelemetry GenAI semantic conventions into a contract that packages are tested against, so whole classes of defect fail CI instead of being reported one issue at a time.
Why
Before this branch, zero of 32 instrumentation packages verified emitted spans against the spec. The 5 packages with
test_semconv*.pyonly assertedCONSTANT == "string.literal"— 441 lines of tautology that never emit or inspect a span, and so cannot catch anything.What it does
open-telemetry/semantic-conventions-genaiat pinned SHA8484f22finto.semconv/, resolved via the pinnedotel/weaver:v0.25.0container. The pin is deliberate: upstream isstability: developmentwith no tags, so trackingmainwould red-CI every package on a third party's merge.gen_ai.*attributes OpenLLMetry emits that upstream does not define, so namespace squatting becomes documented and reversible.Two design corrections made during review
Both were caught by adversarial review and are worth knowing:
required, andgen_ai.response.finish_reasonsis merelyrecommendedin all 9 groups declaring it — so 🐛 Bug Report: [anthropic] Streaming spans drop gen_ai.response.finish_reasons and gen_ai.output.messages for turns with empty content #4362 was undetectable even in enforcing mode. Each package now declares anexpectedset of attributes it promises to emit; a missing promise is a blocking violation.gen_ai.provider.nameisrequiredand lists 16 values, none of which cover ollama/together/replicate — they had no escape (emit → violation, omit → violation). Unknown enum values are now reported but never block.What it already found
A real, shipping, silent data-loss bug in two packages.
groq/span_utils.py:237andbedrock/span_utils.pyreferenceSpanAttributes.GEN_AI_USAGE_CACHE_READ_INPUT_TOKENS, which no longer exists (it moved to upstreamGenAIAttributes). TheAttributeErroris swallowed by@dont_throw, so cache-read token counts silently never reach non-streaming spans. Not fixed here — this branch changes no instrumentation source — but one groq test is markedxfail(strict=True)documenting it.Seven #4362-class gaps, surfaced without a bug report: ollama, mistralai, replicate, alephalpha, writer, together and sagemaker do not emit
gen_ai.response.finish_reasons; langchain emits it on the OpenAI path but not the Anthropic one.What it deliberately does not do
docs/ai/semconv-rollout.md.Known limits
gen_ai.prompt/gen_ai.completionare declared as extensions, but matching is exact and instrumentation emits indexed forms (gen_ai.prompt.0.content). Those still surface as violations; prefix matching is the fix. Documented, not hidden.Reviewing this
_contract/generated.pyis machine-generated — review_contract/_generator.pyinstead..semconv/registry/is vendored upstream — not authored here._contract/extensions.pyis the file that most wants your judgment: each entry is a permanent hole in the contract check, and a few are migrations you may want to prioritise rather than keep.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Tests