Repository navigation
feat: finish v1 leftovers — OTLP tracing, rate-limit rule, verified citations - #22
Merged
Merged
Conversation
…itations - mcp/otlp: OTLP/HTTP-JSON span exporter, transport span hook, broker wiring (BROKER_OTEL_ENDPOINT), compose collector port - MUST/tool-invocations-rate-limited behind --probe-rate-limit; never FAILs - Citations verified against the 2026-07-28 spec source: every MCP/ rule now cites a real heading; fixes three rules that cited a nonexistent basic/security_best_practices page - CI: setup-uv prune-cache false; runbook "Operating the scanner"; docs updates Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
Explain what the 2026-07-28 revision is and what Sentinel checks, contrast with the official conformance suite, and stop implying a PyPI release exists. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
Audited every MCP/ rule against the specification source and the official conformance suite. Withdraws discover-without-negotiated-version (invented), demotes four rules to SHOULD / SENTINEL/STYLE under new ids (old ids are deprecated, never reused), and returns INDETERMINATE instead of FAIL where the wire cannot tell a violation from a permitted behaviour. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
…ERVIEW.md Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
…diagram - `sentinel demo` runs two bundled practice servers and shows both verdicts; fixtures are packaged into the wheel so it works from a plain uvx install - two research-grounded tool-description lint rules - README: 30-second demo, mermaid flow diagram, stack table Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
…neration - benchmarks/official_negatives.py: the MCP maintainers' own broken servers as an answer key (6 of 8 labelled violations caught, misses explained) - benchmarks/sdk_readiness.py: servers built with the official SDKs - reports name the protocol generation; legacy-only servers get a note up front - stub-description threshold 20 -> 12 chars (false positive on "Add two numbers.") Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
…cial suite - legacy-session mode: manifest rules (names, descriptions, injection, schema validity, Origin) run over a previous-revision session instead of reporting "unreachable" - benchmark: 7 of 7 gradable labelled violations caught (was 6 of 8); one fixture is unusable (crashes on every request) and one violation has no rule, both listed - benchmarks/README: head-to-head table against the official suite on an SDK server Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
- StdioTransport: newline-delimited JSON over a child process, same interface as the HTTP transport, so every manifest rule works on the (majority) stdio servers - `sentinel lint --command|--endpoint`: manifest-only rule set, exit 1 on failure - benchmarks/popular_servers.py: 29 published servers, 27 started, 335 tools, 1,184 parameters; 10 of 27 leave parameters undocumented (28%) - benchmarks README: states what could not be compared (Snyk Agent Scan needs cloud access) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
…chart and story-led README - lint rules share one child process (thread-safe stdio transport, borrowed probes); identical results, test pins the start count - benchmarks/chart.py draws the 27-server result as an SVG - README leads with what was found in real servers, then engineering notes with the measured speedup Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
…(28 servers, 17%) The rule counted only schema descriptions. Reading a flagged server's source showed its parameters documented in a docstring Args block, and self-evident names like `path` were flagged. It now accepts prose documentation, nested object descriptions and self-evident plain-string names. Benchmark moves from 28% to 17% of parameters, 10 to 8 servers. Also: registry-scan prints its aggregate as an annotation (no new permissions). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
For the most-failed MUST rules the aggregate now shows the observed HTTP status and JSON-RPC code distribution, to separate "servers are broken" from "servers answer a different code from the one the rule expects". Aggregate only; nothing names a server. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
…ngs; unversioned-served is INDETERMINATE A 4,000-endpoint registry sample showed two false-positive classes. 537 of 572 failures of missing-protocol-version-rejected were "request served", which a server that also serves previous-revision clients is allowed to do. And many failures were HTTP 429/401/5xx answers, where the server refused for a reason unrelated to the rule. Rules now treat those statuses like an unreachable server (INDETERMINATE), and a served unversioned request is INDETERMINATE. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
Full-scan classification stands (15,451 endpoints, 14.6% on the new revision). The rule-level result is superseded: a corrected 4,000-endpoint sample has 16% of new-revision servers passing every verifiable required rule, not 13 of 2,263. The page records how the error was found. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
…on with Cisco mcp-scanner - PRECISION.md: random, seeded, hand-judged samples (parameters 3/12 real before refinement, 7/12 on a fresh draw after; specification findings 15/15 correct) - parameter rule: enums, paging names, flag booleans and unit-suffixed numbers are self-explanatory; quoted parameter names in prose count. Flagged share 28% -> 17% -> 11% - compare_cisco.py + poisoning corpus: Sentinel vs Cisco mcp-scanner (offline YARA) - popular_servers.py can dump tool lists for the comparison Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pdv7R172x4XFq6AxwmmBBF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BROKER_OTEL_ENDPOINT(no arguments or principal in spans)MUST/tool-invocations-rate-limited, opt-in via--probe-rate-limit; never FAILs (PASS if throttled, else INDETERMINATE)MCP/rules now cite real headings. Three security rules cited a page that does not exist and now point at the real security best practices page.docs/CITATIONS.mdand its test removed as obsolete.prune-cache: falsefor the setup-uv post-step flake; runbook section; USING/KIT docsNot changed
Storage split of handles/mrtr/audit: it changes where tenant/principal/expiry and MRTR exactly-once are enforced, so it needs a design decision (see docs/KIT.md).
Test plan
Generated by Claude Code