feat: add AGON playground for real-time encoding and token comparison

Author: Verdenroz

PLAYGROUND.md

@@ -0,0 +1,185 @@
+# AGON Playground Implementation Plan
+
+## Overview
+Add an interactive playground to the MkDocs documentation where users can:
+1. Input JSON data and see real-time encoding in all 4 AGON formats
+2. Compare token counts and savings across formats
+3. See which format would be auto-selected
+4. Edit and experiment with different data shapes
+
+## Technical Approach
+
+### Challenge
+AGON is implemented in Rust with Python bindings - cannot run directly in the browser.
+
+### Solution: Pure JavaScript Implementation
+Implement lightweight JavaScript versions of the encoders for the playground. This provides:
+- Zero latency (no server round-trips)
+- Works offline
+- Self-contained in docs
+- No backend infrastructure needed
+
+The encoding formats are simple string transformations that can be ported accurately to JavaScript.
+
+## Implementation Plan
+
+### 1. Create Playground Page
+**File:** `docs/playground.md`
+
+- New documentation page with embedded playground
+- JSON editor input area
+- Output panels for each format (JSON, Rows, Columns, Struct)
+- Token count comparison chart
+- Auto-format recommendation indicator
+
+### 2. Create JavaScript Encoder Module
+**File:** `docs/javascripts/agon-playground.js`
+
+Implement:
+- `encodeRows(data)` - AGONRows format encoder
+- `encodeColumns(data)` - AGONColumns format encoder
+- `encodeStruct(data)` - AGONStruct format encoder
+- `estimateTokens(text)` - Simple token estimation (chars/4 or GPT tokenizer)
+- `autoSelect(data)` - Format selection logic with min_savings threshold
+
+### 3. Create Playground UI Component
+**File:** `docs/javascripts/playground-ui.js`
+
+Features:
+- **Monaco Editor** for JSON input (VS Code editor with full syntax highlighting, auto-completion, JSON validation)
+- Live encoding preview (debounced for performance)
+- Tab panels for each encoding output
+- Token savings visualization (reuse Chart.js from benchmarks)
+- Copy-to-clipboard buttons
+- Example data dropdown (pre-loaded examples from benchmarks)
+- Error handling for invalid JSON
+
+**Monaco Setup:**
+- Load from CDN: `monaco-editor` via `@monaco-editor/loader`
+- Configure for JSON language with schema validation
+- Theme sync with MkDocs light/dark mode
+
+### 4. Add Styling
+**File:** `docs/stylesheets/playground.css`
+
+- Responsive two-column layout (input | output)
+- Dark/light mode support (matches Material theme)
+- Syntax highlighting for output formats
+- Mobile-friendly collapsible panels
+
+### 5. Update MkDocs Configuration
+**File:** `mkdocs.yml`
+
+- Add playground to navigation
+- Include new JavaScript/CSS files
+- Add extra_css entry
+
+## Files to Create/Modify
+
+| File | Action | Description |
+|------|--------|-------------|
+| `docs/playground.md` | Create | Playground page with HTML structure |
+| `docs/javascripts/agon-playground.js` | Create | Core encoding logic in JS |
+| `docs/javascripts/playground-ui.js` | Create | UI interactions and rendering |
+| `docs/stylesheets/playground.css` | Create | Playground styling |
+| `mkdocs.yml` | Modify | Add nav entry and extra files |
+
+## External Dependencies (CDN)
+
+| Library | Purpose | Size | CDN |
+|---------|---------|------|-----|
+| Monaco Editor | JSON input editor | ~2MB | jsdelivr |
+| js-tiktoken | Accurate token counting | ~300KB | jsdelivr |
+| Chart.js | Already included | - | Already in mkdocs.yml |
+
+## Encoding Implementation Details
+
+### AGONRows Encoder (JS)
+```javascript
+// Input: [{id: 1, name: "Alice"}, {id: 2, name: "Bob"}]
+// Output: [2]{id	name}
+//         1	Alice
+//         2	Bob
+```
+
+Key logic:
+1. Detect if data is uniform array of objects
+2. Extract field names from first object
+3. Generate header `[count]{field1 field2 ...}`
+4. Render each row with tab-delimited values
+5. Handle nested objects with indentation
+
+### AGONColumns Encoder (JS)
+```javascript
+// Input: [{id: 1, name: "Alice"}, {id: 2, name: "Bob"}]
+// Output: [2]
+//         ├ id: 1	2
+//         └ name: Alice	Bob
+```
+
+Key logic:
+1. Transpose data (group by field instead of row)
+2. Generate `[count]` header
+3. Render each field with tree chars (├ / └)
+4. Tab-delimit values within each field line
+
+### AGONStruct Encoder (JS)
+```javascript
+// Input: {price: {fmt: "$100", raw: 100}, change: {fmt: "+5", raw: 5}}
+// Output: @FR: fmt, raw
+//
+//         price: FR($100, 100)
+//         change: FR(+5, 5)
+```
+
+Key logic:
+1. Detect repeated object patterns (same field structure)
+2. Generate template definition if ≥3 occurrences
+3. Replace instances with template calls
+
+### Token Counting (Accurate)
+Use **js-tiktoken** for GPT-accurate token counts:
+- Load `tiktoken` from npm CDN (jsdelivr/unpkg)
+- Use `o200k_base` encoding (GPT-4o default, matches AGON's default)
+- Lazy-load tokenizer on first use to avoid blocking page load
+- Cache tokenizer instance for subsequent calculations
+
+## UI Layout
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  AGON Playground                           [Load Example ▼] │
+├───────────────────────────┬─────────────────────────────────┤
+│  JSON Input               │  Encoded Output                 │
+│  ┌─────────────────────┐  │  ┌───────────────────────────┐  │
+│  │ {                   │  │  │ [JSON] [Rows] [Cols] [St] │  │
+│  │   "users": [        │  │  ├───────────────────────────┤  │
+│  │     {"id": 1, ...   │  │  │ [2]{id  name}             │  │
+│  │   ]                 │  │  │ 1  Alice                  │  │
+│  │ }                   │  │  │ 2  Bob                    │  │
+│  └─────────────────────┘  │  └───────────────────────────┘  │
+│                           │                                 │
+│  ┌─────────────────────────────────────────────────────────┐│
+│  │ Token Comparison Chart                                  ││
+│  │ [====== JSON 45 ======]                                 ││
+│  │ [=== Rows 26 ===] ✓ Best (-42%)                         ││
+│  │ [==== Cols 30 ====]                                     ││
+│  │ [===== Struct 35 =====]                                 ││
+│  └─────────────────────────────────────────────────────────┘│
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Example Data Options
+
+Pre-loaded examples from benchmark data:
+1. User list (simple - Rows wins)
+2. Employee records (wide - Columns wins)
+3. Market data (nested patterns - Struct wins)
+4. Hiking data (mixed - demonstrates auto selection)
+
+## Verification
+
+1. **Unit test encoders**: Compare JS output with Python AGON output for test cases
+2. **Visual testing**: Load playground, test each example, verify outputs match docs
+3. **Build check**: `mkdocs build` succeeds without errors
+4. **Live preview**: `mkdocs serve` shows interactive playground

docs/data/gainers.json

@@ -0,0 +1,34 @@
+{
+  "context": {
+    "task": "Our favorite hikes together",
+    "location": "Boulder",
+    "season": "spring_2025"
+  },
+  "friends": ["ana", "luis", "sam"],
+  "hikes": [
+    {
+      "id": 1,
+      "name": "Blue Lake Trail",
+      "distanceKm": 7.5,
+      "elevationGain": 320,
+      "companion": "ana",
+      "wasSunny": true
+    },
+    {
+      "id": 2,
+      "name": "Ridge Overlook",
+      "distanceKm": 9.2,
+      "elevationGain": 540,
+      "companion": "luis",
+      "wasSunny": false
+    },
+    {
+      "id": 3,
+      "name": "Wildflower Loop",
+      "distanceKm": 5.1,
+      "elevationGain": 180,
+      "companion": "sam",
+      "wasSunny": true
+    }
+  ]
+}