Skip to content

CLAUDE.md

Latest commit

 

History

History
428 lines (332 loc) · 16.9 KB

CLAUDE.md

File metadata and controls

428 lines (332 loc) · 16.9 KB

CLAUDE.md for MicroGrowAgents

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Agent-based system for microbial growth predictions

The project uses uv for dependency management and just as the command runner.

IMPORTANT INSTRUCTIONS

  • we use test driven development, write tests first before implementing a feature
  • do not try and 'cheat' by making mock tests (unless asked)
  • if functionality does not work, keep trying, do not relax the test just to get poor code in
  • always run tests
  • use docstrings

We make heavy use of doctests, these serve as both docs and tests. just test will include these, or do just doctest just to write doctests

In general AVOID try/except blocks, except when these are truly called for, for example when interfacing with external systems. For wrapping deterministic code, these are ALMOST NEVER required, if you think you need them, it's likely a bad smell that your logic is wrong.

Essential Commands

Testing and Quality

  • just test - Run all tests, type checking, and formatting checks
  • just pytest - Run Python tests only
  • just mypy - Run type checking
  • just format - Run ruff linting/formatting checks
  • uv run pytest tests/test_simple.py::test_simple - Run a specific test

Running the CLI

  • uv run MicroGrowAgents --help - Run the CLI tool with options

Experimental Analysis (Dual-Pipeline)

The experimental analysis system supports TWO analysis modes:

Absolute Analysis (Raw OD600):

  • Uses raw OD600 measurements without control subtraction
  • Shows actual biomass achieved in each condition
  • Useful for: identifying highest-performing conditions, comparing to literature values

Relative Analysis (Variation vs Baseline):

  • Calculates fold-change, difference, and percent change vs control baseline
  • Shows improvement over baseline media
  • Useful for: identifying which variations enhance growth, normalizing across experiments

Commands:

  • just analyze-experimental <DATA_DIR> - Run BOTH absolute and relative analyses (recommended)
  • just analyze-experimental-absolute <DATA_DIR> - Run only absolute analysis
  • just analyze-experimental-relative <DATA_DIR> - Run only relative analysis
  • just cluster-experimental <STATS_FILE> <OUTPUT_DIR> <MODE> - Run clustering on results
  • just validate-experimental <SOURCE_DATA_ID> - Validate outputs against schema

Examples:

# Run dual analysis on v10 experimental data (both absolute and relative)
just analyze-experimental data/experimental/plate_designs_v10_maxprooptblock_long__results

# Run only absolute analysis
just analyze-experimental-absolute data/experimental/plate_designs_v10_maxprooptblock_long__results

# Run clustering on absolute results
just cluster-experimental outputs/experimental_analysis_absolute/replicate_statistics_absolute.tsv \\
    outputs/experimental_analysis_clustering_absolute absolute

# Validate all outputs
just validate-experimental plate_designs_v10_maxprooptblock_long__results

Output Directory Naming Schema:

{output_base}/{source_data_id}_experimental_analysis_{mode}/
  • source_data_id: Auto-detected from input directory name (e.g., plate_designs_v10_maxprooptblock_long__results)
  • mode: absolute or relative

Example:

Input:  data/experimental/plate_designs_v10_maxprooptblock_long__results
Output: outputs/plate_designs_v10_maxprooptblock_long__results_experimental_analysis_absolute/

Outputs:

  • outputs/{source_data_id}_experimental_analysis_absolute/ - Absolute mode results
    • {prefix}processed_data_absolute.tsv - Raw OD600 values
    • {prefix}replicate_statistics_absolute.tsv - Summary statistics
  • outputs/{source_data_id}_experimental_analysis_relative/ - Relative mode results
    • {prefix}processed_data_relative.tsv - Fold-change, difference, percent change
    • {prefix}replicate_statistics_relative.tsv - Summary statistics
  • outputs/{source_data_id}_experimental_analysis_clustering_{mode}/ - Clustered heatmaps and cluster assignments

When to use which mode:

  • Use absolute to answer: "Which conditions grew best overall?"
  • Use relative to answer: "Which variations improved over baseline media?"
  • Use both for complete picture: absolute performance + relative improvement

Measurement Types and Interpretation:

OD600 (Optical Density):

  • Absolute mode: Raw OD600 values showing actual biomass
  • Relative mode: Fold-change vs control baseline
  • Higher values = more bacterial growth

Nd_uM (Neodymium / Lanthanide):

  • Values vs baseline media showing relative lanthanide depletion
  • Control baseline (ctrl_media): Functional medium WITH bacterial growth (OD600 ~0.33 at 30h)
  • Initial Nd concentration: 5.5 µM (added to medium)
  • Control bacteria consume most Nd, establishing the baseline for comparison
  • Negative values: More Nd consumption than control baseline (higher bacterial uptake)
  • Positive values: Less Nd consumption than control baseline (lower bacterial uptake)
  • Example: Nd_uM = -0.5 means bacteria consumed 0.5 µM more Nd than growing control
  • Outliers: Values > +10 µM are physically impossible (exceed initial 5.5 µM) and indicate measurement errors
  • Used to assess lanthanide-dependent vs independent growth pathways

Output Validation: The pipeline includes automatic validation against an expected output schema. The validate-experimental command checks that all expected files are present:

Analysis Outputs (16 files per mode):

  • 3 TSV files: control_statistics.tsv, processed_data_{mode}.tsv, replicate_statistics_{mode}.tsv, processed_data_raw.tsv
  • 10 plot files (5 types × 2 formats - PDF + PNG):
    • Growth curves
    • Dose-response curves
    • Heatmaps
    • PCA ingredient space
    • Replicate variability
    • Summary statistics

Clustering Outputs (5 files per mode):

  • CSV: cluster_assignments_growth.csv
  • TXT: cluster_descriptions_growth.txt
  • PDF: clustered_heatmap_growth.pdf, cluster_summary_growth.pdf
  • PNG: clustered_heatmap_growth.png

Validation runs automatically at the end of just analyze-experimental to confirm all outputs are complete.

Response Surface Analysis

The experimental analysis pipeline includes optional response surface modeling using Gaussian Processes to understand ingredient-measurement relationships and multi-objective optimization.

Capabilities:

  • Single-measurement response surfaces (e.g., OD600 alone)
  • Joint multi-objective analysis (e.g., OD600 + Nd_uM)
  • Pareto frontier for identifying optimal trade-offs
  • 3D surface plots and contour maps
  • Predictions over design space

When to use:

  • v13+ designs with variable Neodymium (lanthanide-dependent growth)
  • Multi-measurement datasets (OD600 + Nd_uM)
  • Understanding growth-lanthanide relationships
  • Identifying optimal conditions for multiple objectives

Automatic integration: Response surface analysis runs automatically as part of just analyze-experimental (enabled by default).

Control:

# Disable response surfaces (faster analysis)
python scripts/run_dual_analysis.py <DATA_DIR> --disable-response-surfaces

# Standalone response surface analysis
python scripts/analyze_response_surfaces.py \\
    outputs/{source_data_id}_experimental_analysis_absolute/ \\
    --mode absolute \\
    --measurements OD600 Nd_uM

Outputs (per mode):

  • response_surfaces/surface_predictions_{measurement}_{mode}.csv - Predicted values over design space
  • response_surfaces/surface_3d_{measurement}_{mode}.pdf/png - 3D surface plots
  • response_surfaces/pareto_frontier_{mode}.csv - Pareto-optimal conditions (joint analysis)
  • response_surfaces/pareto_frontier_{mode}.pdf/png - Pareto frontier plot
  • response_surfaces/optimization_report_{mode}.txt - Model parameters and best conditions

Example use case (v13 design): The v13 design varies Neodymium (0-5 µM) to test lanthanide-dependent vs independent growth pathways:

  • High OD600 at low Nd → MxaF-MDH pathway (lanthanide-independent)
  • High OD600 at high Nd → XoxF-MDH pathway (lanthanide-dependent)
  • Pareto frontier identifies conditions optimizing both growth AND Nd uptake

Documentation

Project Architecture

Core Structure

  • src/my_awesome_tool/ - Main package containing the CLI and application logic
    • cli.py - Typer-based CLI interface, entry point for the application
  • tests/ - Test suite using pytest with parametrized tests
  • docs/ - MkDocs-managed documentation with Material theme

Technology Stack

  • Python 3.10+ with uv for dependency management
  • LinkML for data modeling (linkml-runtime)
  • Typer for CLI interface
  • pytest for testing
  • mypy for type checking
  • ruff for linting and formatting
  • MkDocs Material for documentation

Key Configuration Files

  • pyproject.toml - Python project configuration, dependencies, and tool settings
  • justfile - Command runner recipes for common development tasks
  • mkdocs.yml - Documentation configuration
  • uv.lock - Locked dependency versions

Development Workflow

  1. Dependencies are managed via uv - use uv add for new dependencies
  2. All commands are run through just or uv run
  3. The project uses dynamic versioning from git tags
  4. Documentation is auto-deployed to GitHub Pages at https://monarch-initiative.github.io/my-awesome-tool

Project-Specific Files & Locations

Quick Start: Understanding Project State

READ THIS FIRST: docs/STATUS.md

Agentic System Audit

Audit Report: docs/AUDIT_REPORT_BBOP_SKILLS.md (1,579 lines)

  • Comprehensive audit against 9 bbop-skills criteria for local-first agentic systems
  • Overall Compliance: 78% (7/9 PASS)
    • ✅ PASS: Provenance, model tracking, reasoning/code separation, validation, error-correction, RAG
    • ⚠️ PARTIAL: Artifact cleanup (needs documentation/automation)
    • ❌ FAIL: Input data hashing (critical gap), MCP integration (consider adoption)
  • Detailed findings with evidence from codebase analysis
  • Prioritized recommendations with implementation roadmap

Action Checklist: docs/AUDIT_ACTIONS_CHECKLIST.md (880 lines)

  • 7 actionable recommendations with step-by-step implementation
  • Priority levels: 2 HIGH, 3 MEDIUM, 2 LOW
  • Estimated effort and target dates for each action
  • Progress tracking table
  • Current project status and metrics
  • Citation coverage: 90.5% (143/158 DOIs)
  • Recent DOI corrections (7 applied)
  • Remaining issues and next actions
  • Index to all key files

Artifact Management

Cleanup Policy: docs/ARTIFACT_CLEANUP_POLICY.md

  • Three-tier retention model (Archival/Temporary/Ephemeral)
  • Storage estimates: ~185MB steady-state vs 4GB/year unmanaged (96% reduction)
  • Cleanup commands: just clean-old-outputs, just archive-outputs, just clean-ephemeral
  • Automated cleanup via cron (weekly archives, daily ephemeral cleanup)
  • Exception handling and recovery procedures
  • bbop-skills Criterion 9 compliance

Data Integrity:

  • Input data checksums: SHA256 hashes in data/checksums.txt
  • Per-analysis checksums: outputs/*/input_data_checksums.json
  • Verification: just verify-data-integrity
  • bbop-skills Criterion 4 compliance (cryptographic reproducibility)

Key Data Files

Main Dataset: data/raw/mp_medium_ingredient_properties.csv

  • 158 ingredients with 68 columns (47 data + 21 organism context)
  • 158 unique DOI citations tracked
  • LinkML schema: src/microgrowagents/schema/mp_medium_schema.yaml

DOI Corrections

Complete Report: notes/DOI_CORRECTIONS_FINAL_UPDATED.md

  • Comprehensive documentation of all DOI correction work
  • 7 corrections applied (14 CSV instances)
  • Coverage improvement: 86.1% → 90.5%
  • Remaining invalid DOIs and next steps

Correction Definitions: data/corrections/

  • doi_corrections_17_invalid.yaml - Primary correction mappings
  • additional_corrections_found.yaml - Extended research findings
  • doi_corrections.yaml - Historical corrections

Correction Results: data/results/

  • doi_corrections_applied.json - Batch 1 log (4 DOIs)
  • additional_corrections_applied.json - Batch 2 log (3 DOIs)
  • doi_validation_22.json - Validation results

Scripts (Organized by Function)

All scripts located in scripts/ with subdirectories:

DOI Validation: scripts/doi_validation/

  • validate_failed_dois.py - Check DOI HTTP resolution
  • validate_new_corrections.py - Validate correction candidates
  • find_correct_dois.py - Research correct alternatives

DOI Corrections: scripts/doi_corrections/

  • apply_doi_corrections.py - Apply validated corrections
  • apply_additional_corrections.py - Batch corrections
  • clean_invalid_dois.py - Remove invalid DOIs

PDF Downloads: scripts/pdf_downloads/

  • download_all_pdfs_automated.py - Automated PDF retrieval
  • retry_failed_dois_with_fallbackpdf.py - Fallback service

Schema Management: scripts/schema/

  • add_role_columns.py - Add organism context columns
  • migrate_schema.py - Schema migrations

Enrichment: scripts/enrichment/

  • enrich_ingredient_effects.py - Enrich ingredient data

Running Scripts

# Validate DOIs
uv run python scripts/doi_validation/validate_failed_dois.py

# Apply corrections
uv run python scripts/doi_corrections/apply_doi_corrections.py

# Download PDFs
uv run python scripts/pdf_downloads/download_all_pdfs_automated.py

Documentation

Research Notes: notes/

  • 27+ documentation files covering:
    • DOI corrections and validation
    • Citation coverage analysis
    • Schema updates
    • Implementation notes
  • See notes/README.md for guide

Results & Logs: data/results/

  • Validation results (JSON)
  • Download logs (TXT)
  • Processing outputs
  • See data/results/README.md for guide

Current Project Metrics

  • Total DOIs: 158 unique
  • Citation Coverage: 90.5% (143/158 with evidence)
    • PDFs: 92 (58.2%)
    • Abstracts: 44 (27.8%)
    • Missing: 15 (9.5%)
  • DOI Corrections Applied: 7 corrections → 14 CSV cells updated
  • Schema Columns: 68 total (47 data + 21 organism context)
  • Missing Citations: 77 empty DOI cells across 18 ingredients

File Organization

MicroGrowAgents/
├── docs/STATUS.md           # ⭐ START HERE - Current state
├── notes/                   # Research & documentation
├── data/
│   ├── raw/                 # Main CSV and source data
│   ├── corrections/         # DOI correction definitions (YAML/JSON)
│   └── results/             # Validation & processing logs
└── scripts/                 # Organized by function
    ├── doi_validation/
    ├── doi_corrections/
    ├── pdf_downloads/
    ├── enrichment/
    └── schema/

Each directory has a README.md explaining its contents and usage.

Provenance Tracking

Claude Code actions are tracked in .claude/provenance/ for audit trails and analysis:

Session Records

  • Session manifests: .claude/provenance/sessions/YYYY-MM-DD-HH-mm/manifest.yaml

    • Session metadata, goals, results, and summaries
    • Tracked in git for documentation purposes

  • Action logs: .claude/provenance/sessions/YYYY-MM-DD-HH-mm/actions.jsonl

    • Detailed action-by-action logs in JSONL format (one action per line)
    • Gitignored due to size, but queryable with standard tools

  • Human summaries: .claude/provenance/sessions/YYYY-MM-DD-HH-mm/summary.md

    • Human-readable session summaries following notes/SESSION_SUMMARY_*.md pattern
    • Tracked in git for documentation

Usage

Query action logs:

# Count actions by type
cat .claude/provenance/sessions/*/actions.jsonl | jq -r '.type' | sort | uniq -c

# Find all file modifications
jq 'select(.type == "modify")' .claude/provenance/sessions/*/actions.jsonl

# Most frequently accessed files
cat .claude/provenance/sessions/*/actions.jsonl | jq -r '.files_affected[]?' | sort | uniq -c | sort -rn

View session summaries:

# List all sessions
ls -1 .claude/provenance/sessions/

# View session manifest
yq eval . .claude/provenance/sessions/2026-01-10-21-16/manifest.yaml

# Read human summary
cat .claude/provenance/sessions/2026-01-10-21-16/summary.md

See .claude/provenance/README.md for complete documentation, templates, and query examples.

Important: When Working on DOI Corrections

  1. Read docs/STATUS.md for current state
  2. Check notes/DOI_CORRECTIONS_FINAL_UPDATED.md for complete history
  3. Review correction definitions in data/corrections/
  4. Run validation scripts from scripts/doi_validation/
  5. Apply corrections using scripts/doi_corrections/
  6. Check results in data/results/