Skip to content

Commit 76ed6c3

Browse files
CybotTMclaude
CybotTM
and
claude
committed
refactor(audit): modularize audit system with JSON output and version normalization
Refactored monolithic cli_audit.py into modular cli_audit/ package: - audit.py: Main entry point with routing logic - cli_audit/catalog.py: Catalog loading and management - cli_audit/collectors.py: Version collection with parallel execution - cli_audit/detection.py: Local tool detection and version extraction - cli_audit/render.py: Output formatting (table, JSON, colored text) - cli_audit/snapshot.py: Snapshot caching and persistence - cli_audit/tools.py: Tool model and enrichment logic Key features added: - CLI_AUDIT_JSON mode for structured JSON output to shell scripts - normalize_version() function to handle leading/trailing zeros (7.28.00 vs 7.28.0) - Per-tool fresh collection with CLI_AUDIT_COLLECT + CLI_AUDIT_MERGE - Enhanced routing logic for collect/audit/install/upgrade modes This enables: - Shell scripts (guide.sh) to parse audit data reliably - Proper version comparison regardless of zero-padding - Selective snapshot updates without full collection overhead - Integration with install/upgrade workflows 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 432e90c commit 76ed6c3

22 files changed

Lines changed: 4970 additions & 3405 deletions

‎.serena/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/cache

‎.serena/memories/codebase_structure.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# Codebase Structure
2+
3+
## Directory Layout
4+
5+
```
6+
ai_cli_preparation/
7+
├── cli_audit.py # Main audit engine (2,375 lines)
8+
├── cli_audit/ # Phase 2 modular codebase
9+
│ ├── __init__.py # Public API exports
10+
│ ├── common.py # Shared types and utilities
11+
│ ├── config.py # Configuration management
12+
│ ├── environment.py # Environment detection
13+
│ ├── package_managers.py # Package manager abstractions
14+
│ ├── installer.py # Core installation logic
15+
│ ├── bulk.py # Bulk operations
16+
│ ├── reconcile.py # Installation reconciliation
17+
│ ├── upgrade.py # Upgrade workflows
18+
│ ├── breaking_changes.py # Breaking change detection
19+
│ ├── install_plan.py # Installation planning
20+
│ └── logging_config.py # Logging configuration
21+
├── smart_column.py # ANSI/emoji-aware table formatter
22+
├── scripts/ # Installation and utility scripts
23+
│ ├── install_*.sh # Language/tool installers (14 scripts)
24+
│ ├── auto_update.sh # Package manager auto-update
25+
│ ├── guide.sh # Interactive upgrade guidance
26+
│ ├── upgrade_all.sh # Bulk upgrade orchestration
27+
│ ├── lib/ # Shared shell utilities
28+
│ ├── installers/ # Modular installer components
29+
│ └── AGENTS.md # Agent-specific guidance for scripts
30+
├── docs/ # Comprehensive documentation (12 files)
31+
│ ├── INDEX.md # Documentation index by role/task
32+
│ ├── QUICK_REFERENCE.md # One-liner command reference
33+
│ ├── ARCHITECTURE.md # System design and patterns
34+
│ ├── API_REFERENCE.md # Python API documentation
35+
│ ├── DEVELOPER_GUIDE.md # Contribution guidelines
36+
│ ├── TROUBLESHOOTING.md # Common issues and solutions
37+
│ ├── PRD.md # Product Requirements (Phase 2)
38+
│ ├── adr/ # Architecture Decision Records
39+
│ └── phase2_api/ # Phase 2 API specifications
40+
├── tests/ # Test suite (growing)
41+
├── config/ # Configuration examples
42+
├── Makefile # Main task automation
43+
├── Makefile.d/ # Modular make includes
44+
│ ├── user.mk # User-facing commands
45+
│ ├── dev.mk # Development commands
46+
│ └── maint.mk # Maintenance commands
47+
├── latest_versions.json # Committed version cache (offline-first)
48+
├── tools_snapshot.json # Snapshot from last collect
49+
├── pyproject.toml # Python project metadata + tool config
50+
├── package.json # Node dependencies (minimal)
51+
├── AGENTS.md # Root agent guidance (thin, delegates to scoped)
52+
└── README.md # Primary user documentation
53+
```
54+
55+
## Key Files and Responsibilities
56+
57+
### Core Audit Engine
58+
- **cli_audit.py**: Main entry point, 50+ tool definitions, parallel execution logic
59+
- **smart_column.py**: Table formatting with ANSI/emoji preservation
60+
- **TOOLS**: Tuple of Tool dataclasses defining name, candidates, upstream source
61+
62+
### Phase 2 Modular Architecture
63+
- **cli_audit/__init__.py**: Public API surface for programmatic use
64+
- **cli_audit/common.py**: Shared types (InstallResult, Environment, etc.)
65+
- **cli_audit/config.py**: YAML config loading and validation
66+
- **cli_audit/installer.py**: Tool installation with package manager abstraction
67+
- **cli_audit/bulk.py**: Parallel bulk install/upgrade/reconcile operations
68+
- **cli_audit/upgrade.py**: Upgrade candidate detection and execution
69+
- **cli_audit/breaking_changes.py**: Semver analysis for breaking changes
70+
- **cli_audit/reconcile.py**: Duplicate installation cleanup
71+
72+
### Installation Scripts
73+
- **scripts/install_python.sh**: Python via uv (preferred) or pyenv
74+
- **scripts/install_node.sh**: Node.js via nvm (preferred) or binary
75+
- **scripts/install_rust.sh**: Rust via rustup
76+
- **scripts/install_core.sh**: Core tools (fd, fzf, ripgrep, jq, yq, bat, delta, just)
77+
- **scripts/auto_update.sh**: Detect and update all package managers
78+
- **scripts/guide.sh**: Interactive upgrade wizard with remediation
79+
80+
### Data Files
81+
- **latest_versions.json**: Committed cache with manual overrides + lookup hints
82+
- **tools_snapshot.json**: Generated by `make update`, consumed by `make audit`
83+
- **.env**: Local environment overrides (gitignored)
84+
- **.env.default**: Default environment variables (committed)
85+
86+
## Code Organization Patterns
87+
88+
### Tool Definition Pattern
89+
```python
90+
@dataclass(frozen=True)
91+
class Tool:
92+
name: str # Display name
93+
candidates: tuple[str, ...] # Executable names to search
94+
source_kind: str # gh|pypi|crates|npm|gnu|skip
95+
source_args: tuple[str, ...] # (owner, repo) for gh, (package,) for pypi
96+
category: str # For grouping (search, security, etc.)
97+
homepage: str | None # Documentation URL
98+
```
99+
100+
### Parallel Execution Pattern
101+
```python
102+
with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
103+
futures = [executor.submit(audit_tool, tool) for tool in TOOLS]
104+
for future in as_completed(futures):
105+
result = future.result()
106+
```
107+
108+
### Lock Ordering Enforcement
109+
```python
110+
# Always acquire in this order to prevent deadlock:
111+
with MANUAL_LOCK: # First: manual cache lock
112+
with HINTS_LOCK: # Second: hints cache lock
113+
# Safe update logic
114+
```
115+
116+
## Module Dependencies
117+
- cli_audit.py: Standalone, no internal imports
118+
- cli_audit/* modules: Import from cli_audit.common, avoid circular deps
119+
- scripts/*.sh: Source from scripts/lib/*.sh for shared utilities

‎.serena/memories/known_issues_and_notes.md‎

Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,225 @@
1+
# Known Issues and Notes
2+
3+
## Version Discrepancies
4+
5+
### glab (GitLab CLI)
6+
**Issue**: Cached latest version may be outdated
7+
**Details**:
8+
- User reported latest version is 1.74.0 (from https://gitlab.com/gitlab-org/cli/-/releases)
9+
- Cache may have older version (1.22 mentioned)
10+
- **Action needed**: Update cache entry for glab when running `make update`
11+
12+
**Resolution:**
13+
```bash
14+
# Manual update in latest_versions.json
15+
# Or run update to fetch latest from GitHub releases
16+
make update
17+
18+
# Verify
19+
make audit-glab
20+
```
21+
22+
## System-Specific Notes
23+
24+
### Operating System
25+
- **Platform**: Linux
26+
- **System**: WSL2 (Windows Subsystem for Linux)
27+
- **Kernel**: 6.6.87.2-microsoft-standard-WSL2
28+
- **Current Date**: 2025-10-23
29+
30+
### Python Environment
31+
- **Version**: Python 3.10+ required (Python 3.14.0rc2 currently available)
32+
- **Package Management**: uv preferred over pipx/pip
33+
- **Virtual Environments**: Optional, not required for core audit
34+
35+
### Git Configuration
36+
- **Current Branch**: main
37+
- **Default Branch**: main (for PRs)
38+
- **Commit Style**: Conventional Commits (type(scope): description)
39+
40+
## WSL2-Specific Considerations
41+
42+
### File System
43+
- **Line Endings**: LF enforced (EditorConfig)
44+
- **Git Config**: May need `git config --global core.autocrlf input`
45+
- **Permissions**: chmod may behave differently than native Linux
46+
47+
### Network
48+
- **WSL2 uses NAT**: May affect network detection
49+
- **DNS**: Sometimes needs manual configuration
50+
- **Proxy**: Check if corporate proxy affects GitHub/PyPI access
51+
52+
### Tool Availability
53+
- **Some tools may need manual install**: Not all package managers work identically in WSL2
54+
- **Docker**: May need Docker Desktop with WSL2 integration
55+
- **Snap**: May not work in WSL2, prefer apt/brew alternatives
56+
57+
## Cache Management Notes
58+
59+
### Lock Ordering
60+
**Critical**: Always acquire locks in this order to prevent deadlock:
61+
1. MANUAL_LOCK (manual cache updates)
62+
2. HINTS_LOCK (lookup hints updates)
63+
64+
**Never acquire in reverse order!**
65+
66+
### Atomic Writes
67+
All cache updates use atomic file operations:
68+
1. Write to temporary file
69+
2. Rename to final location
70+
3. Prevents corruption from interrupted writes
71+
72+
### Lookup Hints
73+
`latest_versions.json` contains `__hints__` key with working upstream methods:
74+
- Speeds up subsequent runs
75+
- Safe to edit or remove (will rebuild)
76+
- Format: `"tool_name": "gh"` or `"tool_name": "pypi"`, etc.
77+
78+
## Common Troubleshooting
79+
80+
### Version Detection Fails
81+
**Symptoms**: Tool shows as NOT INSTALLED but is in PATH
82+
**Causes**:
83+
- Non-standard version flag
84+
- Tool doesn't support --version or -v
85+
- Version output parsing fails
86+
87+
**Solution**:
88+
```bash
89+
# Debug single tool
90+
CLI_AUDIT_DEBUG=1 python3 cli_audit.py --only <tool>
91+
92+
# Check manually
93+
which <tool>
94+
<tool> --version
95+
<tool> -v
96+
<tool> version
97+
```
98+
99+
### Network Timeouts
100+
**Symptoms**: Empty upstream versions, slow updates
101+
**Causes**:
102+
- Slow network connection
103+
- GitHub rate limiting
104+
- Upstream service temporarily down
105+
106+
**Solutions**:
107+
```bash
108+
# Increase timeout
109+
CLI_AUDIT_TIMEOUT_SECONDS=10 make update
110+
111+
# Use offline mode
112+
make audit-offline
113+
114+
# Check rate limit
115+
curl -I https://api.github.com/rate_limit
116+
117+
# Use GitHub token to increase rate limit
118+
export GITHUB_TOKEN=ghp_...
119+
make update
120+
```
121+
122+
### Cache Corruption
123+
**Symptoms**: Invalid JSON errors, wrong versions
124+
**Solution**:
125+
```bash
126+
# Nuclear option: delete and regenerate
127+
rm latest_versions.json tools_snapshot.json
128+
make update
129+
make audit
130+
```
131+
132+
### Parallel Execution Issues
133+
**Symptoms**: Deadlocks, race conditions, inconsistent results
134+
**Solutions**:
135+
```bash
136+
# Reduce workers
137+
CLI_AUDIT_MAX_WORKERS=4 make update
138+
139+
# Sequential execution for debugging
140+
CLI_AUDIT_MAX_WORKERS=1 make update
141+
142+
# Check lock ordering in code changes
143+
```
144+
145+
## Performance Notes
146+
147+
### Benchmark Targets
148+
- **make update**: 3-10s for 50+ tools (parallel, MAX_WORKERS=16)
149+
- **make audit**: <100ms (snapshot render)
150+
- **make audit-offline**: <200ms (with hints)
151+
- **Single tool**: <50ms
152+
153+
### Optimization Tips
154+
- Use snapshot-based workflow (separate collect/render)
155+
- Enable FAST_MODE for development: `CLI_AUDIT_FAST=1`
156+
- Disable unnecessary features: `CLI_AUDIT_TIMINGS=0`, `CLI_AUDIT_PROGRESS=0`
157+
- Increase MAX_WORKERS if more CPU cores: `CLI_AUDIT_MAX_WORKERS=32`
158+
159+
## Development Environment
160+
161+
### Recommended Tools for Development
162+
- **pyflakes**: Linting (make lint)
163+
- **mypy**: Type checking (optional, make lint-types)
164+
- **black**: Formatting (optional, make format)
165+
- **isort**: Import sorting (optional, make format)
166+
- **shellcheck**: Shell script linting
167+
- **jq**: JSON manipulation and validation
168+
169+
### Optional Setup
170+
```bash
171+
# Install development dependencies
172+
pip install -e ".[dev]"
173+
174+
# Or with uv
175+
uv pip install -e ".[dev]"
176+
177+
# Enable direnv (optional)
178+
direnv allow
179+
```
180+
181+
### Editor Configuration
182+
- **EditorConfig**: .editorconfig present (4 spaces, LF, UTF-8, trim trailing)
183+
- **VSCode**: Respect EditorConfig, install Python extension
184+
- **Vim/Neovim**: Use editorconfig plugin
185+
- **IntelliJ/PyCharm**: EditorConfig support built-in
186+
187+
## Future Work (Phase 2 Complete, Phase 3 Planned)
188+
189+
### Phase 2 Status (Complete)
190+
- [x] Context-aware installation management
191+
- [x] Breaking change detection
192+
- [x] Installation reconciliation
193+
- [x] Bulk operations
194+
- [x] Configuration file support
195+
- [x] Upgrade workflows
196+
197+
### Phase 3 (Planned)
198+
- [ ] Advanced dependency resolution
199+
- [ ] Rollback mechanisms
200+
- [ ] Integration tests
201+
- [ ] CI/CD pipeline
202+
- [ ] Performance benchmarks
203+
- [ ] Additional package managers
204+
205+
See [docs/PRD.md](docs/PRD.md) for comprehensive Phase 2 specifications.
206+
207+
## Contributing Notes
208+
209+
### PR Size Guidelines
210+
- **Ideal**: ~≤300 net LOC changed
211+
- **Rationale**: Easier to review, faster to merge, lower risk
212+
- **Large changes**: Break into multiple PRs when possible
213+
214+
### Review Process
215+
1. Self-review checklist (see task_completion_checklist.md)
216+
2. CI checks pass (when added)
217+
3. Documentation updated in same PR
218+
4. No secrets or credentials committed
219+
5. Conventional commit messages
220+
6. Linked to issue/ticket if applicable
221+
222+
### Communication
223+
- **Issues**: GitHub Issues for bugs, features, discussions
224+
- **PRs**: Clear description, link to issue, changelog section
225+
- **Commits**: Self-documenting with Conventional Commits format

‎.serena/memories/project_overview.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Project Overview
2+
3+
## Project Purpose
4+
AI CLI Preparation is a tool version auditing and installation management utility specifically designed for AI coding agents (like Claude Code). It ensures AI agents have access to 50+ essential developer CLI tools with up-to-date versions.
5+
6+
**Core Functionality:**
7+
- Detects installed versions of developer tools across PATH
8+
- Compares local versions against latest upstream releases (GitHub, PyPI, crates.io, npm)
9+
- Reports version status with actionable remediation hints
10+
- Provides installation/upgrade scripts for missing or outdated tools
11+
- Supports offline-first operation with committed version caches
12+
13+
**Architecture Phases:**
14+
- **Phase 1 (Complete)**: Tool detection, version auditing, offline-first caching, snapshot-based workflows
15+
- **Phase 2 (Complete)**: Context-aware installation/upgrade management, breaking change detection, reconciliation
16+
- **Phase 3 (Future)**: Advanced dependency resolution, rollback mechanisms
17+
18+
## Key Characteristics
19+
- **Agent-focused**: Tool selection optimized for AI coding agent needs (ripgrep, fd, jq, ast-grep, etc.)
20+
- **Offline-capable**: Committed cache (`latest_versions.json`) enables offline audits
21+
- **Snapshot-based**: Separate collect (`make update`) and render (`make audit`) phases
22+
- **Parallel execution**: ThreadPoolExecutor with 16 workers for fast collection (~3-10s for 50+ tools)
23+
- **Zero dependencies**: Core audit uses Python standard library only (no pip install needed)
24+
- **Comprehensive**: Covers 50+ tools across multiple categories (runtimes, search, security, formatters, cloud/infra)
25+
26+
## Primary Use Cases
27+
1. **AI agent readiness verification**: Confirm coding agents have necessary tools before starting work
28+
2. **Development environment setup**: Bootstrap new machines with agent-optimized toolchains
29+
3. **Version compliance checking**: Ensure tools meet minimum version requirements
30+
4. **Automated toolchain updates**: Keep development tools current via upgrade workflows
31+
5. **Installation method reconciliation**: Identify and resolve duplicate tool installations
32+
33+
## Tech Stack
34+
- **Language**: Python 3.9+ (primary language, standard library only for core)
35+
- **Task Automation**: GNU Make (Makefile + modular includes in Makefile.d/)
36+
- **Installation Scripts**: Bash (with set -euo pipefail, shellcheck-compliant)
37+
- **Data Formats**: JSON for caching (latest_versions.json, tools_snapshot.json)
38+
- **Dependencies**:
39+
- Core: Python standard library only
40+
- Optional: packaging, PyYAML (for Phase 2 config support)
41+
- Dev: pytest, flake8, mypy, black, isort (listed in pyproject.toml)
42+
43+
## Project Status
44+
- **Production-ready**: Phase 1 detection/audit fully functional and tested in production
45+
- **Beta**: Phase 2 installation management (2.0.0-alpha.6)
46+
- **Documentation**: Comprehensive (12 docs files, 189KB, including architecture, API reference, troubleshooting)
47+
- **Testing**: Smoke tests exist (test_smoke.sh), formal test suite acknowledged as needed in README

0 commit comments

Comments
 (0)