refactor: consolidate to 2-file data model

Simplify from 4 JSON files to 2:
- upstream_versions.json: latest available versions (committed)
- local_state.json: machine-specific state (gitignored)

Removed:
- latest_versions.json: merged into upstream_versions.json
- tools_snapshot.json: now gitignored (generated on demand)

Updated all documentation references from latest_versions.json
to upstream_versions.json.

Note: docs still reference __hints__ feature which was removed
during prior refactoring - needs separate cleanup pass.

Author: CybotTM

.gitignore

@@ -34,6 +34,7 @@ htmlcov/
 
 # Machine-specific audit state (Phase 2.1 split files)
 local_state.json
+tools_snapshot.json
 
 # Node.js
 node_modules/

README.md

@@ -730,24 +730,17 @@ make audit
 - Some package managers (like Go) don't have built-in bulk update mechanisms - manual updates are required
 - The script gracefully handles missing package managers (skips them)
 
-## Caching
+## Data Files
 
-- Manual baseline (committed): `latest_versions.json` in this repo (override with `CLI_AUDIT_MANUAL_FILE`). Used as the primary source in offline mode; also used as a fallback when online lookups fail. Example content:
+The audit system uses two JSON files:
 
-```json
-{
-  "rust": "1.89.0",
-  "jq": "jq-1.8.1",
-  "parallel": "20240322"
-}
-```
-
-- Auto-updates: when an online lookup succeeds, the tool writes the discovered latest value back into `latest_versions.json` (toggle with `CLI_AUDIT_WRITE_MANUAL=0`).
-- Offline behavior: set `CLI_AUDIT_OFFLINE=1` to use `latest_versions.json` exclusively.
-
-### Lookup hints
+| File | Purpose | Git Tracked |
+|------|---------|-------------|
+| `upstream_versions.json` | Latest available versions from upstream sources | Yes (committed) |
+| `local_state.json` | Machine-specific installed tool versions | No (gitignored) |
 
-To speed up future runs, the audit records which upstream retrieval method worked last per tool. These hints are stored inside `latest_versions.json` under the special key `"__hints__"`. They help prioritize the fastest working method on subsequent runs and are safe to edit or remove; they will be rebuilt.
+- **Offline mode**: Set `CLI_AUDIT_OFFLINE=1` to use cached `upstream_versions.json` without network calls
+- **Baseline refresh**: Run `python audit.py --update-baseline` to update upstream versions
 
 ## License
 MIT

docs/ARCHITECTURE.md

@@ -206,13 +206,13 @@ sleep_time = base * (2 ** attempt) + random.uniform(0, jitter)
 
 **Cache Hierarchy (fastest → slowest):**
 
-1. **Hints Cache** (`__hints__` in `latest_versions.json`)
+1. **Hints Cache** (`__hints__` in `upstream_versions.json`)
    - Stores which API method worked last per tool
    - Example: `"gh:BurntSushi/ripgrep": "latest_redirect"`
    - Purpose: Skip failed methods, try successful ones first
    - Lock: `HINTS_LOCK` (acquired after `MANUAL_LOCK`)
 
-2. **Manual Cache** (`latest_versions.json`)
+2. **Manual Cache** (`upstream_versions.json`)
    - Committed to repository
    - Used in offline mode (`CLI_AUDIT_OFFLINE=1`)
    - Updated on successful upstream fetches (unless `CLI_AUDIT_WRITE_MANUAL=0`)
@@ -227,7 +227,7 @@ sleep_time = base * (2 ** attempt) + random.uniform(0, jitter)
 **Lock Ordering Rule:**
 ```python
 with MANUAL_LOCK:
-    # Update latest_versions.json
+    # Update upstream_versions.json
     with HINTS_LOCK:
         # Update __hints__ section
 ```
@@ -239,8 +239,8 @@ with MANUAL_LOCK:
 
 ```python
 # Global locks for cache writes
-MANUAL_LOCK = threading.Lock()  # For latest_versions.json
-HINTS_LOCK = threading.Lock()   # For __hints__ in latest_versions.json (nested)
+MANUAL_LOCK = threading.Lock()  # For upstream_versions.json
+HINTS_LOCK = threading.Lock()   # For __hints__ in upstream_versions.json (nested)
 
 # ThreadPoolExecutor for parallel tool audits
 with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
@@ -369,7 +369,7 @@ Attempt 1: Try upstream API (with retries)
   ↓ (on failure)
 Attempt 2: Check hints cache for alternative method
   ↓ (on failure)
-Attempt 3: Use manual cache (latest_versions.json)
+Attempt 3: Use manual cache (upstream_versions.json)
   ↓ (on failure)
 Result: Mark as UNKNOWN, continue audit
 ```

docs/ARCHITECTURE_DIAGRAM.md

@@ -486,18 +486,18 @@ get_available_version(tool, pm, cache_ttl)
     │   ├─→ SUCCESS:
     │   │   ├─→ Update in-memory cache
     │   │   ├─→ Update hints cache
-    │   │   └─→ Write to latest_versions.json
+    │   │   └─→ Write to upstream_versions.json
     │   │
     │   └─→ FAILURE:
-    │       └─→ Fallback to latest_versions.json
+    │       └─→ Fallback to upstream_versions.json
     │           └─→ Return cached or "UNKNOWN"
     │
     └─→ Returns: version string
 
 Multi-Tier Cache Hierarchy:
 1. In-memory (fastest)
 2. Hints (method preference)
-3. Manual cache (latest_versions.json)
+3. Manual cache (upstream_versions.json)
 4. Snapshot (tools_snapshot.json)
 ```
 
@@ -511,7 +511,7 @@ Rule: Always acquire MANUAL_LOCK before HINTS_LOCK
 Prevents: Deadlocks
 
 with MANUAL_LOCK:
-    # Update latest_versions.json
+    # Update upstream_versions.json
     with HINTS_LOCK:
         # Update __hints__ section
 ```
@@ -606,7 +606,7 @@ Example: Version lookup failure
 └──────────────────────────────────────┘
     │
     ├─→ No network requests
-    ├─→ Use committed cache (latest_versions.json)
+    ├─→ Use committed cache (upstream_versions.json)
     ├─→ Use snapshot (tools_snapshot.json)
     └─→ Baseline version checking only
 ```

docs/CLI_REFERENCE.md

@@ -142,7 +142,7 @@ make audit   # Render only
 ### Offline Mode
 
 ```bash
-# Use only manual cache (latest_versions.json)
+# Use only manual cache (upstream_versions.json)
 CLI_AUDIT_OFFLINE=1 python3 cli_audit.py
 
 # Offline + render from snapshot
@@ -190,7 +190,7 @@ CLI_AUDIT_OFFLINE=1 CLI_AUDIT_RENDER=1 python3 cli_audit.py
 | Variable | Type | Default | Description |
 |----------|------|---------|-------------|
 | `CLI_AUDIT_SNAPSHOT_FILE` | path | `tools_snapshot.json` | Snapshot file path |
-| `CLI_AUDIT_MANUAL_FILE` | path | `latest_versions.json` | Manual cache path |
+| `CLI_AUDIT_MANUAL_FILE` | path | `upstream_versions.json` | Manual cache path |
 | `CLI_AUDIT_WRITE_MANUAL` | bool | `1` | Auto-update manual cache |
 | `CLI_AUDIT_MANUAL_FIRST` | bool | `0` | Try manual cache before network |
 
@@ -708,7 +708,7 @@ CLI_AUDIT_COLLECT=1 python3 cli_audit.py
 jq '.' tools_snapshot.json
 
 # Rebuild from scratch
-rm tools_snapshot.json latest_versions.json
+rm tools_snapshot.json upstream_versions.json
 make update
 ```
 

docs/DEPLOYMENT.md

@@ -517,7 +517,7 @@ make audit-offline
 
 **What Changes:**
 - No upstream API calls (GitHub, PyPI, crates.io, npm)
-- Uses `latest_versions.json` exclusively for version data
+- Uses `upstream_versions.json` exclusively for version data
 - Marks upstream method as `manual`
 - Displays `(offline)` in readiness summary
 
@@ -532,12 +532,12 @@ make audit-offline
 1. **Update manual cache online:**
 ```bash
 make update
-# Populates latest_versions.json with current data
+# Populates upstream_versions.json with current data
 ```
 
 2. **Commit manual cache:**
 ```bash
-git add latest_versions.json
+git add upstream_versions.json
 git commit -m "chore: update manual version cache"
 ```
 
@@ -548,7 +548,7 @@ CLI_AUDIT_OFFLINE=1 python3 cli_audit.py --only python
 
 ### Offline Cache Management
 
-**Cache File:** `latest_versions.json` (override with `CLI_AUDIT_MANUAL_FILE`)
+**Cache File:** `upstream_versions.json` (override with `CLI_AUDIT_MANUAL_FILE`)
 
 **Structure:**
 ```json
@@ -610,7 +610,7 @@ CLI_AUDIT_PROGRESS=0
 
 # Paths
 CLI_AUDIT_SNAPSHOT_FILE=tools_snapshot.json
-CLI_AUDIT_MANUAL_FILE=latest_versions.json
+CLI_AUDIT_MANUAL_FILE=upstream_versions.json
 
 # Cache
 CLI_AUDIT_WRITE_MANUAL=1
@@ -728,7 +728,7 @@ jobs:
         uses: actions/cache@v3
         with:
           path: tools_snapshot.json
-          key: tools-snapshot-${{ runner.os }}-${{ hashFiles('latest_versions.json') }}
+          key: tools-snapshot-${{ runner.os }}-${{ hashFiles('upstream_versions.json') }}
           restore-keys: |
             tools-snapshot-${{ runner.os }}-
 
@@ -746,7 +746,7 @@ jobs:
         if: always()
         with:
           path: tools_snapshot.json
-          key: tools-snapshot-${{ runner.os }}-${{ hashFiles('latest_versions.json') }}
+          key: tools-snapshot-${{ runner.os }}-${{ hashFiles('upstream_versions.json') }}
 ```
 
 #### PR Comment Workflow

docs/DEVELOPER_GUIDE.md

@@ -54,7 +54,7 @@ make update && make audit
 ### 5. Commit and Push
 
 ```bash
-git add cli_audit.py latest_versions.json
+git add cli_audit.py upstream_versions.json
 git commit -m "feat(tools): add support for new-tool"
 git push -u origin feature/add-new-tool
 ```
@@ -143,7 +143,7 @@ CLI_AUDIT_JSON=1 python3 cli_audit.py --only your-tool | jq '.'
 
 ### Step 6: Update Manual Cache
 
-Add initial version to `latest_versions.json`:
+Add initial version to `upstream_versions.json`:
 
 ```json
 {
@@ -185,7 +185,7 @@ python3 cli_audit.py --only deno | python3 smart_column.py -s "|" -t
 
 **4. Commit:**
 ```bash
-git add cli_audit.py latest_versions.json
+git add cli_audit.py upstream_versions.json
 git commit -m "feat(tools): add Deno runtime support"
 ```
 
@@ -251,7 +251,7 @@ When updating caches, always use locks:
 ```python
 # ALWAYS acquire MANUAL_LOCK before HINTS_LOCK
 with MANUAL_LOCK:
-    # Update latest_versions.json
+    # Update upstream_versions.json
     set_manual_latest(tool_name, version)
 
     with HINTS_LOCK:
@@ -313,7 +313,7 @@ CLI_AUDIT_DEBUG=1 CLI_AUDIT_TRACE=1 python3 cli_audit.py 2>&1 | grep "slow"
 
 ```bash
 # Ensure manual cache is valid JSON
-jq '.' latest_versions.json > /dev/null
+jq '.' upstream_versions.json > /dev/null
 
 # Ensure snapshot is valid
 jq '.__meta__.schema_version' tools_snapshot.json
@@ -418,10 +418,10 @@ CLI_AUDIT_DEBUG=1 python3 cli_audit.py --only problematic-tool 2>&1 | tee debug.
 
 ```bash
 # View manual cache
-jq '.' latest_versions.json
+jq '.' upstream_versions.json
 
 # View hints
-jq '.__hints__' latest_versions.json
+jq '.__hints__' upstream_versions.json
 
 # View snapshot metadata
 jq '.__meta__' tools_snapshot.json
@@ -457,7 +457,7 @@ test(smoke): verify JSON output schema
 - [ ] Code passes `pyflakes` lint
 - [ ] Manual test passed for affected tools
 - [ ] Smoke test passed (`bash scripts/test_smoke.sh`)
-- [ ] Updated `latest_versions.json` if adding tools
+- [ ] Updated `upstream_versions.json` if adding tools
 - [ ] Updated documentation if changing behavior
 - [ ] Commit message follows conventions
 

docs/FUNCTION_REFERENCE.md

@@ -25,7 +25,7 @@ Functions for reading/writing snapshot files that decouple data collection from
 Load snapshot from file.
 
 **Parameters:**
-- `paths` (`Sequence[str] | None`) - Custom paths to try (default: `[SNAPSHOT_FILE, "latest_versions.json"]`)
+- `paths` (`Sequence[str] | None`) - Custom paths to try (default: `[SNAPSHOT_FILE, "upstream_versions.json"]`)
 
 **Returns:**
 - `dict[str, Any]` - Document with `__meta__` and `tools` keys
@@ -543,7 +543,7 @@ Multi-tier caching system for offline operation and performance.
 
 ### `load_manual_versions() -> None`
 
-Load manual cache from `latest_versions.json`.
+Load manual cache from `upstream_versions.json`.
 
 **Side Effects:** Populates global `MANUAL_VERSIONS` dict
 
@@ -591,12 +591,12 @@ Update manual cache with new version.
 - `tool_name` (`str`) - Tool name
 - `tag_or_version` (`str`) - Version string or tag
 
-**Side Effects:** Writes to `latest_versions.json` (requires `MANUAL_LOCK`)
+**Side Effects:** Writes to `upstream_versions.json` (requires `MANUAL_LOCK`)
 
 **Usage:**
 ```python
 set_manual_latest("ripgrep", "14.1.1")
-# Updates latest_versions.json atomically
+# Updates upstream_versions.json atomically
 ```
 
 **See:** [API_REFERENCE.md#set_manual_latest](API_REFERENCE.md#set_manual_latest)
@@ -605,7 +605,7 @@ set_manual_latest("ripgrep", "14.1.1")
 
 ### `load_hints() -> None`
 
-Load API method hints from `__hints__` in `latest_versions.json`.
+Load API method hints from `__hints__` in `upstream_versions.json`.
 
 **Side Effects:** Populates global `HINTS` dict
 
@@ -656,7 +656,7 @@ Store API method hint for future runs.
 - `key` (`str`) - Hint key
 - `value` (`str`) - Method that worked
 
-**Side Effects:** Writes to `__hints__` in `latest_versions.json`
+**Side Effects:** Writes to `__hints__` in `upstream_versions.json`
 
 **Lock Ordering:** Requires `MANUAL_LOCK` → `HINTS_LOCK`
 
@@ -1152,7 +1152,7 @@ See [API_REFERENCE.md#environment-variables](API_REFERENCE.md#environment-variab
 
 - **[API_REFERENCE.md](API_REFERENCE.md)** - Complete function signatures
 - **Data Structures** - Tool dataclass, TOOLS registry
-- **File Schemas** - tools_snapshot.json, latest_versions.json
+- **File Schemas** - tools_snapshot.json, upstream_versions.json
 
 ### Developer Guide
 

docs/MIGRATION_GUIDE.md

@@ -80,7 +80,7 @@ cli_audit/
 ### File Locations
 
 **Unchanged:**
-- `latest_versions.json` (cache)
+- `upstream_versions.json` (cache)
 - `tools_snapshot.json` (snapshot)
 - `smart_column.py` (formatting)
 - `Makefile` and `Makefile.d/`

docs/PRD.md

@@ -58,7 +58,7 @@ AI CLI Preparation is a specialized environment audit and installation managemen
 - Multi-source API support (GitHub, PyPI, crates.io, npm, GNU FTP)
 - HTTP layer with retries, exponential backoff, rate limiting
 - Per-origin request caps (GitHub: 5/min, PyPI: 10/min, crates.io: 5/min)
-- Offline-first design with committed cache (latest_versions.json)
+- Offline-first design with committed cache (upstream_versions.json)
 
 **Output Formats:**
 - Table view with status icons (✓ UP-TO-DATE, ↑ OUTDATED, ✗ NOT INSTALLED, ? UNKNOWN)
@@ -79,8 +79,8 @@ AI CLI Preparation is a specialized environment audit and installation managemen
 - Independent tool audits (failures isolated)
 
 **Cache Hierarchy:**
-- **Hints**: Optimization hints for faster lookups (__hints__ in latest_versions.json)
-- **Manual**: User-committed versions (latest_versions.json)
+- **Hints**: Optimization hints for faster lookups (__hints__ in upstream_versions.json)
+- **Manual**: User-committed versions (upstream_versions.json)
 - **Upstream**: Live API queries (fallback)
 
 **Resilience Patterns:**