Document AI agent skill-usage telemetry#1229
Open
IEvangelist wants to merge 2 commits into
Open
Conversation
Extend the Microsoft-collected CLI telemetry page to cover the agent telemetry hooks installed by 'aspire agent init': the three event types (skill_invocation, tool_invocation, reference_file_read), the exact low-cardinality fields recorded, the privacy guarantees (only Aspire-owned skill/tool names and skill-relative reference paths; never absolute paths, repo or user names, file contents, or tool arguments), and the ASPIRE_CLI_AGENT_TELEMETRY_OPTOUT / --no-telemetry-hooks opt-out paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Frontend HTML artifact readyThe latest frontend build uploaded the This comment updates automatically when a new frontend build artifact is uploaded. |
Remove the AI-only opt-out section (ASPIRE_CLI_AGENT_TELEMETRY_OPTOUT) and the 'aspire agent init --no-telemetry-hooks' note. The single ASPIRE_CLI_TELEMETRY_OPTOUT switch disables all CLI telemetry including AI agent skill usage. Soften the over-claimed opt-out re-check wording. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the Aspire CLI telemetry reference documentation to include the new AI agent skill-usage telemetry that’s emitted via the aspire agent init-installed PostToolUse hook and recorded through the existing CLI telemetry pipeline.
Changes:
- Adds an “AI agent skill usage” subsection under Scope describing how the
PostToolUsehook forwards usage events toaspire agent telemetry. - Adds a new Data points entry (v13.5) and a dedicated section documenting the three event types and the low-cardinality fields recorded.
- Documents privacy guarantees and clarifies that
ASPIRE_CLI_TELEMETRY_OPTOUTdisables this telemetry as well.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
12
to
+18
| Aspire CLI telemetry is collected when using the CLI to run commands. Telemetry is gathered during normal CLI usage, unless you have [opted out](#how-to-opt-out) of telemetry collection. | ||
|
|
||
| ### AI agent skill usage | ||
|
|
||
| When you run `aspire agent init`, Aspire installs Aspire's AI skills and a `PostToolUse` hook into each detected agent client's user-level configuration (for example, GitHub Copilot CLI and Claude Code). The hook lets Aspire understand which of its AI skills and tools are actually used so the team can invest in the most valuable ones. | ||
|
|
||
| After each agent tool use, the hook inspects the event and—only when an **Aspire** skill, Aspire MCP tool, or Aspire skill reference file was involved—forwards a low-cardinality usage event to the hidden `aspire agent telemetry` command, which records it through the same CLI telemetry pipeline. Tool uses that don't involve Aspire skills or tools are ignored. See [AI agent skill usage data](#ai-agent-skill-usage-data) for exactly what's recorded. |
| - `tool_invocation` — an Aspire MCP tool was called. | ||
| - `reference_file_read` — a reference file bundled alongside an Aspire skill was read. | ||
|
|
||
| Each event records only the following low-cardinality, Aspire-owned values: |
alistairmatthews
approved these changes
Jul 1, 2026
alistairmatthews
left a comment
alistairmatthews
left a comment
Contributor
There was a problem hiding this comment.
Please apply my suggestion. Otherwise, good to go.
|
|
||
| When you run `aspire agent init`, Aspire installs Aspire's AI skills and a `PostToolUse` hook into each detected agent client's user-level configuration (for example, GitHub Copilot CLI and Claude Code). The hook lets Aspire understand which of its AI skills and tools are actually used so the team can invest in the most valuable ones. | ||
|
|
||
| After each agent tool use, the hook inspects the event and—only when an **Aspire** skill, Aspire MCP tool, or Aspire skill reference file was involved—forwards a low-cardinality usage event to the hidden `aspire agent telemetry` command, which records it through the same CLI telemetry pipeline. Tool uses that don't involve Aspire skills or tools are ignored. See [AI agent skill usage data](#ai-agent-skill-usage-data) for exactly what's recorded. |
Contributor
There was a problem hiding this comment.
This sentence is long and confusing.
Suggested change
| After each agent tool use, the hook inspects the event and—only when an **Aspire** skill, Aspire MCP tool, or Aspire skill reference file was involved—forwards a low-cardinality usage event to the hidden `aspire agent telemetry` command, which records it through the same CLI telemetry pipeline. Tool uses that don't involve Aspire skills or tools are ignored. See [AI agent skill usage data](#ai-agent-skill-usage-data) for exactly what's recorded. | |
| After each agent tool use, the hook checks if an **Aspire** skill, Aspire MCP tool, or Aspire skill reference file was involved. If so, the hook inspects the event and forwards a low-cardinality usage event to the hidden `aspire agent telemetry` command. The command records the event through the same CLI telemetry pipeline. Tool uses that don't involve Aspire skills or tools are ignored. See [AI agent skill usage data](#ai-agent-skill-usage-data) for exactly what's recorded. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the Microsoft-collected CLI telemetry reference page to document the AI agent skill-usage telemetry installed by
aspire agent init:PostToolUsehook.skill_invocation,tool_invocation,reference_file_read) and the exact low-cardinality fields recorded.ASPIRE_CLI_TELEMETRY_OPTOUTswitch disables all CLI telemetry, including AI agent skill usage.English source only; translations are handled by the localization workflow.
The implementation shipped in microsoft/aspire#18009 (merged to
main, Aspire 13.5); the canonical hook scripts live in microsoft/aspire-skills#30. This doc has been verified against the merged behavior — the documented event types and recorded fields match the tags the shippedaspire agent telemetrycommand emits.Part of microsoft/aspire#18008