tok/CITATION.cff at main · GrayCodeAI/tok · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
cff-version: 1.2.0
title: "Tok: A Go Library for Prompt Compression, Output Filtering, Token Estimation, and Secret Detection"
message: "If you use Tok in your research, please cite it using these metadata."
type: software
authors:
  - name: "GrayCode AI"
    website: "https://github.com/lakshmanpatel"
repository-code: "https://github.com/GrayCodeAI/tok"
url: "https://github.com/GrayCodeAI/tok"
license: MIT
version: "0.1.0"
date-released: "2026-05-14"
keywords:
  - token-reduction
  - llm
  - ai-coding-assistant
  - compression
  - context-window
  - bpe
  - token-estimation
  - secret-detection
  - go-library
abstract: >-
  Tok is a pure Go library (no CGO, no CLI) that provides prompt compression,
  output filtering, BPE token estimation, cost calculation, and secret
  detection for AI coding agents and other LLM workloads. It exposes a
  31-layer compression pipeline (entropy, LLMLingua-style perplexity
  pruning, AST preservation, gisting, H2O, attention-sink, ChunkKV, and
  more) and ships a 33-pattern secret scanner with optional Shannon-entropy
  analysis. Built on research from 50+ papers, it achieves 60-90% token
  reduction on common development operations. The library is consumed by
  hawk, eyrie, and yaad in the hawk-eco ecosystem, and is available to any
  Go program via `go get github.com/GrayCodeAI/tok`.
references:
  - type: article
    title: "Selective Context for Language Models"
    authors:
      - family-names: "Li"
        given-names: "Yucheng"
    year: 2023
    notes: "Foundation for Layer 1: Entropy Filtering"
  - type: article
    title: "LLMLingua: Compressing Prompts for Accelerated Inference"
    authors:
      - family-names: "Jiang"
        given-names: "Huiqiang"
    year: 2023
    notes: "Foundation for Layer 2: Perplexity Pruning"
  - type: article
    title: "LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios"
    authors:
      - family-names: "Jiang"
        given-names: "Huiqiang"
    year: 2024
    notes: "Foundation for Layer 5: Contrastive Ranking"
  - type: article
    title: "H2O: Heavy-Hitter Oracle for Efficient Generative Inference"
    year: 2023
    notes: "Foundation for Layer 13: H2O Filter"
  - type: article
    title: "Efficient Streaming Language Models with Attention Sinks"
    year: 2023
    notes: "Foundation for Layer 14: Attention Sink"