Collect AI agent skill-usage telemetry (CLI command + agent init hooks)#18009
Conversation
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 18009Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 18009" |
This comment was marked as low quality.
This comment was marked as low quality.
Sorry, something went wrong.
|
❓ CLI E2E Tests unknown — 113 passed, 0 failed, 2 unknown (commit View all recordings
📹 Recordings uploaded automatically from CI run #27236204071 |
There was a problem hiding this comment.
Pull request overview
This PR adds opt-out, low-cardinality usage telemetry for Aspire's own AI skills and MCP tools (part of #18008), mirroring microsoft/azure-skills. It introduces a hidden aspire agent telemetry CLI command that records a single reported activity per agent PostToolUse event, and auto-registers PostToolUse hooks during aspire agent init for detected GitHub Copilot CLI and Claude Code clients. Privacy is enforced by recording only allowlisted, Aspire-owned identifiers (skill/tool names, skill-relative reference path, session GUID); the single ASPIRE_CLI_TELEMETRY_OPTOUT switch gates everything, re-checked by both the hook scripts and the command.
Changes:
- New hidden
aspire agent telemetrycommand that validates/normalizes hook values, drops anything invalid (emitting no span when nothing survives), suppresses the genericaspire/cli/mainspan, force-flushes the reported (Azure Monitor) provider, and always exits 0 so a hook can never break the agent. aspire agent initPhase 6 materializestrack-telemetry.{sh,ps1}to~/.aspire/hooksand writes user-levelPostToolUsehooks via a conservative, idempotent JSON merge that never clobbers malformed/unexpected configs.- Embedded hook scripts + bundle sync/verify tooling, new localized strings, and comprehensive unit/script tests.
Reviewed changes
Copilot reviewed 50 out of 51 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
src/Aspire.Cli/Commands/AgentTelemetryCommand.cs |
New hidden command; validates/drops values, emits one reported span, always exits 0. |
src/Aspire.Cli/Telemetry/AgentTelemetryInvocation.cs |
Null-safe leading-token matcher for agent telemetry used to suppress the main span. |
src/Aspire.Cli/Telemetry/TelemetryManager.cs |
Adds ForceFlushReportedAsync + bounded flush constant for reliable one-shot export. |
src/Aspire.Cli/Program.cs |
Wires opt-out, main-span suppression, reported-flush, and DI registrations. |
src/Aspire.Cli/Commands/AgentInitCommand.cs |
Phase 6 best-effort telemetry hook configuration; surfaces configured/skipped clients. |
src/Aspire.Cli/Commands/McpInitCommand.cs |
Threads the new configurator through the legacy delegating command. |
src/Aspire.Cli/Agents/Hooks/TelemetryHookConfigurator.cs |
Writes Copilot/Claude user-level hooks; conservative idempotent JSON merge + atomic write. |
src/Aspire.Cli/Agents/Hooks/TelemetryHookInstaller.cs |
Materializes embedded scripts with LF/no-BOM, atomic write, executable bit. |
src/Aspire.Cli/Agents/Hooks/HookCommandFormatter.cs + interfaces/records |
Quoting helpers and DI contracts for hook wiring. |
src/Aspire.Cli/Agents/Hooks/track-telemetry.{sh,ps1} |
Embedded hook scripts: classify events, honor opt-out, always emit {"continue":true}. |
src/Aspire.Cli/Aspire.Cli.csproj |
Embeds the hook scripts with matching LogicalNames. |
src/Aspire.Cli/Resources/AgentCommandStrings.{resx,Designer.cs} + xlf/* |
Adds 8 localized strings (command/option descriptions, hook install/skip messages). |
eng/scripts/aspire-skills-bundle.common.ps1 + update/verify scripts |
Bundle sync/verify of embedded hooks with SHA-256 integrity against a pinned commit. |
tests/Aspire.Cli.Tests/Commands/AgentTelemetryCommandTests.cs |
Verifies tag emission, value dropping, unsafe-path rejection, exit-0 behavior. |
tests/Aspire.Cli.Tests/Telemetry/AgentTelemetryInvocationTests.cs |
Verifies the leading-token matcher across positive/negative/null cases. |
tests/Aspire.Cli.Tests/Telemetry/TelemetryConfigurationTests.cs |
Verifies opt-out gating of Azure Monitor for the agent telemetry path. |
tests/Aspire.Cli.Tests/Commands/AgentInitCommandTests.cs |
Verifies hook installation for a detected client; adds FakeDetectingDetector. |
tests/Aspire.Cli.Tests/Utils/CliTestHelper.cs |
Registers the new hook services and AgentTelemetryCommand for tests. |
Files not reviewed (1)
- src/Aspire.Cli/Resources/AgentCommandStrings.Designer.cs: Generated file
IEvangelist
force-pushed
the
dapine/agent-telemetry
branch
from
4299a9b to
9a213e4
Compare
This comment has been minimized.
This comment has been minimized.
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
IEvangelist
force-pushed
the
dapine/agent-telemetry
branch
from
9a213e4 to
6857fad
Compare
This comment has been minimized.
This comment has been minimized.
IEvangelist
force-pushed
the
dapine/agent-telemetry
branch
from
6857fad to
78c5732
Compare
This comment has been minimized.
This comment has been minimized.
IEvangelist
force-pushed
the
dapine/agent-telemetry
branch
from
78c5732 to
e2489fa
Compare
This comment has been minimized.
This comment has been minimized.
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
IEvangelist
force-pushed
the
dapine/agent-telemetry
branch
from
e2489fa to
0891f58
Compare
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
…lemetry Introduces a hidden 'aspire agent telemetry' subcommand that receives parsed agent PostToolUse values (event type, client, session, skill/tool name, file reference, timestamp) and writes them into Aspire's reported telemetry pipeline. - AgentTelemetryInvocation detector suppresses the main CLI usage span and the first-run notice so each hook event emits only one agent_telemetry span. - Per-field validation/sanitization drops unsafe or overlong values (absolute paths, traversal, unknown event types) so only low-cardinality Aspire-owned identifiers are recorded. The command never throws and always exits 0. - Multi-level opt-out: global ASPIRE_CLI_TELEMETRY_OPTOUT and new AI-only ASPIRE_CLI_AGENT_TELEMETRY_OPTOUT, honored before any provider is created. - Bounded reported-provider force-flush for the one-shot fire-and-forget path. - Localized command/option strings + unit tests (41 passing). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…init'
Materialize the track-telemetry.{sh,ps1} hook scripts to ~/.aspire/hooks and
register a PostToolUse hook into each detected client's user-level config
(GitHub Copilot CLI ~/.copilot/hooks/aspire-telemetry.json and Claude Code
~/.claude/settings.json). Default-on for parity with azure-skills, with a
--no-telemetry-hooks opt-out; transmission stays gated by the telemetry
opt-out env vars that both the scripts and the 'agent telemetry' command
re-check.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Simplify the AI agent skill-usage telemetry to honor only the existing
ASPIRE_CLI_TELEMETRY_OPTOUT switch. Remove the AI-specific
ASPIRE_CLI_AGENT_TELEMETRY_OPTOUT environment variable and the
'aspire agent init --no-telemetry-hooks' option; hooks are always installed
during 'agent init' and whether telemetry is sent stays gated by the single
opt-out, re-checked at runtime by both the hook scripts and the CLI command.
Also address code-review findings:
- Emit no reported span when no value survives validation (was a tagless span).
- Add TelemetryHookScriptTests that run the real embedded track-telemetry.{sh,ps1}
scripts end-to-end and assert the resulting 'agent telemetry' invocation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
JsonNode.Parse(content)?.AsObject() only guarded against JsonException, but AsObject() throws InvalidOperationException when the settings.json root is valid JSON that isn't an object (e.g. [1,2,3], "str", 42). That escaped both the configurator and the best-effort callers and would crash 'aspire agent init'. Parse first, then pattern-match: null starts a fresh object, a JsonObject is used as-is, and any other root is skipped as UnexpectedConfigShape. Adds a regression test covering a non-object (array) root. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Close the remaining test gaps for the telemetry hook scripts and make the
aspire-skills bundle CI responsible for the embedded hook scripts too.
Tests (tests/Aspire.Cli.Tests/Agents/TelemetryHookScriptTests.cs):
- Add 3 bash + 4 pwsh cases covering previously untested branches:
SKILL.md read -> skill_invocation, reference_file_read relative-path
forwarding, Claude `aspire:` skill-name prefix stripping, and VS Code
`mcp_aspire_` MCP client detection.
Bundle CI (Option C, commit-pinned source sync):
- eng/scripts/aspire-skills-bundle.common.ps1: new shared helper used by
both update and verify so their hook logic cannot drift. Fetches hooks
from microsoft/aspire-skills at the release's immutable commit and hashes
over LF-normalized UTF-8 (no BOM) so .ps1 (text=auto) and .sh (eol=lf)
hash consistently regardless of checkout line endings.
- update-aspire-skills-bundle.ps1: pin to the release commit, sync
hooks/scripts/track-telemetry.{sh,ps1} into src/Aspire.Cli/Agents/Hooks,
and record { commitSha, files: { name: sha256 } } in the bundle metadata.
Releases that predate the hooks warn and skip (transition-friendly).
- verify-aspire-skills-bundle.ps1: when metadata records hooks, cross-check
that each embedded file matches the recorded hash AND that the recorded
hash matches the canonical source at the pinned commit.
- verify-aspire-skills-bundle.yml: trigger on the embedded hook scripts and
the new shared helper so hand-edits are verified.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Configure the PostToolUse hooks per the GitHub Copilot CLI and Claude Code hooks references: - Copilot keeps shell form; the powershell value invokes pwsh (PowerShell 7+), the documented Windows prerequisite, instead of Windows PowerShell 5.1. - Claude moves to exec form (command + args) as the docs recommend for path-referencing hooks, avoiding the Git Bash shell hop and PowerShell-style quoting mismatch on Windows. Uses pwsh on Windows and bash on Unix. - IsAspireHook matches the script name in either command (legacy shell form) or args (exec form) so re-init stays idempotent across the format change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make the PostToolUse hook scripts fail-safe by construction rather than relying on every code path remembering to print the response: - bash: a single EXIT trap is now the one guaranteed emit point for the continue response; all paths just exit 0. A final explicit exit 0 stops the CLI's exit code (e.g. timeout's 124) leaking through as the hook's exit code. - pwsh: a top-level trap is the last-resort net for any unhandled terminating error, so the redundant per-call try/catch around stdin read, JSON parse, and CLI invoke are removed. - Both: a fast pre-filter returns immediately when the raw payload contains no skill/aspire token, skipping all sed/grep (bash) and JSON parsing (pwsh) on the common non-Aspire event. Validated: existing pwsh script tests pass; the bash script and both fail-safe paths (missing CLI, malformed payload) verified to emit exactly one response and exit 0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
IEvangelist
force-pushed
the
dapine/agent-telemetry
branch
from
7b1ee7c to
aecbb0b
Compare
Tests selector (audit mode)The full test matrix and all jobs still run in audit mode. The tests and jobs below are what selective CI would run under enforcement. Runs the full test matrix + all jobs (ALL) — run-all fallback: 'eng/scripts/aspire-skills-bundle.common.ps1' is neither Layer-1-owned nor matched by a Layer 2 rule Selection computed for commit |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Re-syncs hooks/scripts/track-telemetry.{sh,ps1} byte-for-byte (LF-normalized)
with the canonical embedded copies in microsoft/aspire#18009 after a round of
hardening:
- Parse Copilot CLI's JSON-encoded-string toolArgs (not just nested objects) so
skill and reference-file detection actually fires for the Copilot client.
- Guaranteed single hook response via an EXIT trap (sh) / top-level trap (ps1).
- Bound the Aspire CLI call (sh: timeout 10; ps1: child process WaitForExit) so a
hung or slow CLI can't stall the agent's tool loop.
- Case-insensitive opt-out parity between the two scripts.
- Fast-path early return for the common non-Aspire PostToolUse event.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
IEvangelist
requested review from
JamesNK,
davidfowl and
mitchdenny
as code owners
|
❌ CI Failure Analysis: Code Issue Detected The CI build failed due to compile errors caused by changes in this PR. The failing file is Failure details:
The CI will not be automatically rerun. Please fix the compile errors and push an updated commit.
|
|
❌ CI Failure Analysis: Code Issue Detected The CI build failed due to compilation errors caused by changes in this PR. Failure details:
The CI will not be automatically rerun. Please fix the compilation errors in
|
|
❌ CI Failure Analysis: Code Issue Detected The CI build failed due to compilation errors caused by changes in this PR. Failure details:
Errors: Likely cause: The PR modifies The CI will not be automatically rerun. Please fix the compilation errors and push an updated commit.
|
|
❌ CI Failure Analysis: Code Issue Detected The CI build failed due to compilation errors caused by changes in this PR. Failure details:
The CI will not be automatically rerun. Please fix the type mismatches in
|
|
❌ CI Failure Analysis: Code Issue Detected The CI build failed due to compilation errors caused by changes in this PR. Failure details:
Compilation errors: Likely cause: The PR modified The CI will not be automatically rerun. Please fix the type mismatches in
|
davidfowl
left a comment
davidfowl
left a comment
There was a problem hiding this comment.
Please test this well after the merge
|
Documentation changes were written and committed to branch Triggered signals (5): Intended docs changes (needed but not merged):
See the workflow run for details: https://github.com/microsoft/aspire/actions/runs/28602209839 |
Description
Adds opt-out, low-cardinality usage telemetry for Aspire's own AI skills and MCP tools, mirroring the approach used by
microsoft/azure-skills.Two parts:
aspire agent telemetrycommand — receives the parsed hook values and records a singleaspire/cli/agent_telemetryreported activity through the existing CLI telemetry pipeline. It honors opt-out before creating any provider, suppresses the automaticaspire/cli/mainspan (so each hook event is one event, not an inflated command span), force-flushes the reported provider for reliable one-shot export, validates/normalizes every value (emitting no span when nothing valid survives), and always exits 0 so a hook can never break the agent.aspire agent init— materializes thetrack-telemetry.{sh,ps1}scripts to~/.aspire/hooksand writes aPostToolUsehook into each detected supported client's user-level config (GitHub Copilot CLI~/.copilot/hooks/aspire-telemetry.jsonand Claude Code~/.claude/settings.json). Hooks are always installed, for parity with azure-skills. Conservative JSON merge (never clobbers malformed/unexpected configs), idempotent, AOT-safe.Privacy: only Aspire-owned identifiers are recorded (allowlisted skill/tool names, the skill-relative reference path, a session GUID) — never absolute paths, repo/user names, file contents, or tool arguments.
Opt-out: a single switch,
ASPIRE_CLI_TELEMETRY_OPTOUT, disables all CLI telemetry including AI agent skill usage. It's checked by the hook scripts (a fast short-circuit) and honored by the command before anything is sent.Tests include
TelemetryHookScriptTests, which run the real embeddedtrack-telemetry.{sh,ps1}scripts end-to-end and assert the resultingagent telemetryinvocation.Draft pending coordinated review with the companion PRs (hook scripts in microsoft/aspire-skills#30, docs in microsoft/aspire.dev#1229). Clients shipped now: Copilot CLI + Claude Code; VS Code / OpenCode hook schemas are deferred. See the tracking issue for known edge cases.
Part of #18008
Checklist