Commit f583731
committed
Add token counting utilities and update documentation
Features:
- Add benchmark dependency group with tiktoken>=0.4.0 to pyproject.toml
- Export count_tokens, estimate_savings, and compare_formats utilities
- Implement token counting using tiktoken with o200k_base encoding (gpt5/gpt5-mini)
Documentation Updates:
- Add Token Counting & Comparison section to main README with examples
- Update docs/README.md with new utility functions in API reference list
- Add roadmap section announcing planned comprehensive benchmarks
- Add complete Utility Functions section to docs/api.md covering:
* count_tokens() - Token counting with tiktoken
* estimate_savings() - JSON vs TOON comparison metrics
* compare_formats() - Formatted comparison tables
- Add Token Efficiency examples with cost estimation patterns
- Update LLM integration guide with Measuring Token Savings section
- Include cost calculation examples and integration patterns
- Update model references from GPT-4 to gpt5 throughout docs
- Add benchmark disclaimer noting comprehensive benchmarks coming soon
Technical Details:
- Update tokenizer documentation from GPT-4o/GPT-4 to gpt5/gpt5-mini
- Fix TypedDict usage examples in docs/api.md (EncodeOptions uses dict syntax)
- Clarify DecodeOptions is a class while EncodeOptions is a TypedDict
- Add toon-spec/ submodule files (CHANGELOG.md and SPEC.md v1.3)1 parent d249dff commit f583731
6 files changed
Lines changed: 417 additions & 24 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| 13 | + | |
| 14 | + | |
13 | 15 | | |
14 | 16 | | |
15 | | - | |
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
| |||
72 | 73 | | |
73 | 74 | | |
74 | 75 | | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
75 | 104 | | |
76 | 105 | | |
77 | 106 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
27 | 30 | | |
28 | 31 | | |
29 | 32 | | |
| |||
53 | 56 | | |
54 | 57 | | |
55 | 58 | | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
56 | 68 | | |
57 | 69 | | |
58 | 70 | | |
| |||
95 | 107 | | |
96 | 108 | | |
97 | 109 | | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
98 | 132 | | |
99 | 133 | | |
100 | 134 | | |
| |||
0 commit comments