feat(mcp): wire MCP tools to real tok package and align docs to library-only#58
Merged
Merged
Conversation
…ry-only Three changes that together bring tok's MCP surface and its docs in line with the actual library-only codebase. mcp/server.go — the existing implementation was a stub that hard-coded: - count_tokens: len(text)/4 (no BPE) - estimate_cost: return 0.0 (no pricing) - compress_text: collapseWS (no pipeline) Replace each handler with a real call into the public tok API: - count_tokens(text, model) → tok.EstimateTokens / EstimateTokensForModel - estimate_cost(model, in, out) → tok.GetModelPricing × (in/1000*ip + out/1000*op) - compress_text(text, mode, budget?) → tok.Compress with preset Option - redact_secrets(text, entropy?) → tok.NewSecretDetector (new tool) Add a 4th tool redact_secrets backed by the 33-pattern secret detector. The legacy collapseWS helper is removed. mcp/server_test.go — replace the stub tests with real-pipeline tests. The 4x token reduction assertion on aggressive mode is the stub-regression guard: a future refactor that re-introduces len/4 / 0.0 / collapseWS shortcuts fails CI. ARCHITECTURE.md — still claimed 'Go library and CLI tool' and drew a Tok CLI box with command runner / filter selector / rewrite engine. Also referenced agents/ and hooks/ directories that don't exist, and a 'Command Rewriting' flow that was never library-side. Rewrite to library-only architecture, drop the CLI box and the Command Rewriting section, update the file structure table to match reality, update Security to 33 patterns, update Build & Release to library distribution via 'go get'. CITATION.cff — title and abstract claimed 'Token-Aware CLI Proxy'. Update to 'A Go Library for Prompt Compression, Output Filtering, Token Estimation, and Secret Detection' and reword the abstract to describe the library API, the 31-layer pipeline, and the go-get distribution path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three changes that together bring tok's MCP surface and its docs in line with the actual library-only codebase.
mcp/server.go
The existing implementation was a stub that hard-coded:
count_tokens:len(text)/4(no BPE)estimate_cost: returns0.0(no pricing)compress_text:collapseWS(no pipeline)Replace each handler with a real call into the public tok API:
count_tokens(text, model)→tok.EstimateTokens/EstimateTokensForModelestimate_cost(model, in, out)→tok.GetModelPricing×(in/1000*ip + out/1000*op)compress_text(text, mode, budget?)→tok.Compresswith presetOptionredact_secrets(text, entropy?)→tok.NewSecretDetector(new tool)The legacy
collapseWShelper is removed.mcp/server_test.go
Replace the stub tests with real-pipeline tests. The 4× token reduction assertion on aggressive mode is the stub-regression guard: a future refactor that re-introduces
len/4/0.0/collapseWSshortcuts fails CI. 10 new tests added.ARCHITECTURE.md
Still claimed 'Go library and CLI tool' and drew a Tok CLI box with command runner / filter selector / rewrite engine. Also referenced
agents/andhooks/directories that don't exist, and a 'Command Rewriting' flow that was never library-side.Rewrite to library-only architecture, drop the CLI box and the Command Rewriting section, update the file structure table to match reality, update Security to 33 patterns, update Build & Release to library distribution via
go get.CITATION.cff
Title and abstract claimed 'Token-Aware CLI Proxy'. Update to 'A Go Library for Prompt Compression, Output Filtering, Token Estimation, and Secret Detection' and reword the abstract to describe the library API, the 31-layer pipeline, and the go-get distribution path.
Verification
go build ./...✓go test ./mcp/ -race -count=1✓ 16/16 passgo test .(main package smoke) ✓gofumpt -l mcp//goimports -l mcp/✓ clean