Skip to content

Add MCP project profile advisor#356

Open
ozymandiashh wants to merge 1 commit into
getagentseal:mainfrom
ozymandiashh:codex/mcp-profile-advisor
Open

Add MCP project profile advisor#356
ozymandiashh wants to merge 1 commit into
getagentseal:mainfrom
ozymandiashh:codex/mcp-profile-advisor

Conversation

@ozymandiashh
Copy link
Copy Markdown
Contributor

Summary

  • Adds an MCP project profile advisor to codeburn optimize.
  • Flags MCP servers that are clearly useful in one or two hot projects but are also loaded into cold projects where they are never invoked.
  • Estimates cold-project schema overhead with cache-aware accounting and recommends project-scoping instead of blindly removing the server.

Why

Existing MCP optimize findings answer two narrower questions:

  1. Is an MCP server configured but never used?
  2. Is a large MCP server loaded with mostly unused tools?

This adds a third question: is the MCP useful, but scoped too broadly?

For example, a GitHub MCP server may be useful in the main code repo but unnecessary in docs/playground repos. Removing it globally would hurt the hot workflow; project-scoping it preserves the useful setup while reducing cold-project schema overhead.

What changed

  • Adds detectMcpProfileAdvisor() to src/optimize.ts.
  • Builds per-server, per-project profile stats from existing parsed data:
    • loaded sessions come from session.mcpInventory
    • invocations come from session.mcpBreakdown
  • Emits a finding when:
    • a server is loaded in at least three projects
    • at least 80% of invocations are concentrated in the top one or two hot projects
    • at least two cold sessions loaded the server without invoking it
  • Suppresses servers already eligible for the existing low MCP tool coverage finding to avoid duplicate reports.
  • Uses combined cache-aware token accounting when multiple candidate servers share the same cold sessions, so they cannot independently claim the same cache bucket.

Validation

I validated the behavior by running the real detectMcpProfileAdvisor() export against controlled ProjectSummary fixtures via npx tsx --eval. This constructs the actual edge cases the feature is meant to handle and prints detector output.

{
  "one_hot_project_profile": {
    "title": "1 MCP server should be project-scoped",
    "tokensSaved": 4000,
    "expectedTokensSaved": 4000,
    "containsHotProject": true,
    "containsColdProjects": true,
    "proof": "two cold projects each loaded 4 github MCP tools: 4*400 schema tokens * 1.25 cache-write pricing * 2 cold sessions = 4,000 effective tokens"
  },
  "two_hot_project_profile": {
    "title": "1 MCP server should be project-scoped",
    "containsBothHotProjects": true,
    "containsColdProjects": true,
    "proof": "top two projects can form the hot profile while cold projects are still recommended for project-scoping"
  },
  "shared_cold_session_cap": {
    "title": "2 MCP servers should be project-scoped",
    "tokensSaved": 5000,
    "expectedTokensSaved": 5000,
    "proof": "github + slack share the same two cold calls; combined schema budget is capped by each 2,000-token cache bucket before 1.25 cache-write pricing, not summed independently"
  },
  "healthy_even_use_guard": {
    "finding": null,
    "proof": "server used evenly across all loaded projects emits no profile finding"
  },
  "low_tool_coverage_suppression": {
    "finding": null,
    "proof": "server already eligible for low MCP tool coverage emits no duplicate profile finding"
  }
}

What this proves:

  • A server hot in one project and cold in two projects emits the profile finding with the expected 4,000 effective-token estimate.
  • The hot profile can be one or two projects, so shared real usage is not misclassified as global waste.
  • Multiple candidate MCP servers sharing the same cold session are capped once at the combined call cache bucket: 5,000 tokens, not the inflated independent sum.
  • Evenly used servers emit no finding.
  • Servers already owned by low MCP tool coverage emit no duplicate profile finding.

Supporting checks:

  • ./node_modules/.bin/tsc --noEmit --pretty false
  • npx vitest run tests/mcp-coverage.test.ts — 29 tests passed
  • npm run build
  • npm test -- --run — 62 files / 877 tests passed
  • git diff --check origin/main...HEAD
  • Claude Opus 4.7, effort max review:
    • first pass found multi-candidate cold-session token overstatement
    • after fixing combined accounting, final review returned PASS
  • Gemini 3.1 Pro Preview review returned PASS on the amended diff

Notes

  • The recommendation is intentionally a prompt, not an automatic config edit. Users should confirm the profile before changing MCP scope.
  • This PR does not depend on Add MCP and skill ROI optimize insights #354; it is based on current origin/main and uses existing MCP inventory/breakdown data.

@ozymandiashh ozymandiashh marked this pull request as ready for review May 18, 2026 23:21
@ozymandiashh
Copy link
Copy Markdown
Contributor Author

@iamtoruk quick meta question so we do not create noise: are PRs like #356 / #357 in the direction you want for CodeBurn contributions, or would you prefer issues/design notes before feature PRs?

Happy to keep changes small, evidence-backed, and wait for feedback before opening more exploratory optimize/features work.

@iamtoruk
Copy link
Copy Markdown
Member

Hey, appreciate you asking.

For new features like #356 and #357, please open an issue first so we can discuss the idea before any code. We try to keep CodeBurn focused and add features based on user demand rather than exploring every possible direction. That way nobody wastes time building something we end up not shipping.

Bug fixes and small improvements against existing functionality are always welcome as direct PRs.

@ozymandiashh
Copy link
Copy Markdown
Contributor Author

ozymandiashh commented May 19, 2026

Makes sense, thanks for clarifying.

I will open issues/design notes first for new feature ideas going forward. For #356 / #357, happy to either leave them as concrete references for discussion or close them and reopen the ideas as issues, whichever is cleaner for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants