Skip to content

Latest commit

 

History

History
490 lines (399 loc) · 25.8 KB

File metadata and controls

490 lines (399 loc) · 25.8 KB

Developer Guide

Run tests

pip install -e ".[dev]"
python -m pytest -q

On Windows, if pytest cannot access the default temp directory, use a workspace-local base temp directory:

New-Item -ItemType Directory -Force .codex_tmp\pytest-basetemp | Out-Null
python -m pytest -q --basetemp .codex_tmp\pytest-basetemp

Run smoke test

bash scripts/smoke_test.sh

PowerShell:

.\scripts\smoke_test.ps1

Adding a new CLI command

  1. Add implementation in a focused module under src/openrepro/.
  2. Add a Typer command in cli.py.
  3. Add tests under tests/.
  4. Update README and docs.
  5. Ensure generated artifacts are timestamped or safely overwritten only when appropriate.

Artifact manifests

Demo and sweep runs must write manifest.json after all run artifacts are complete. When adding a new run command, define its required artifacts in artifact_manager.py, write the manifest at the end of the command, and add validation tests.

PDF and candidate extraction

PDF ingestion uses pdfplumber. Extracted text and page provenance should stay under workspace/extracted_sources/, and all generated formulas or parameters must remain marked as candidates until a human verifies them.

v0.8.0 candidate extraction should preserve evidence provenance: source path, chunk index, page number when available, context windows, and evidence quality signals. These fields help review candidates; they are not verification results.

v1.1.0 candidate extraction should also preserve section labels, caption anchors, and risk flags. Risk flags are triage hints for human review, not validation results or automated rejection decisions.

API provider extensions

Real providers should be opt-in. They must write actual usage records and must not fabricate tokens or costs.

v0.4.0 ships MockProvider plus a minimal OpenAI-compatible provider path. Real calls must require explicit configuration, environment-backed secrets, and request-hash cache accounting. Do not store API keys in project files.

v0.5.2 adds provider cache policy and preview redaction. New provider code should write redacted prompt_preview and response_preview fields to usage records, store cache files under provider/model/task namespaces, and respect cache_enabled, cache_ttl_seconds, and redact_prompts.

Benchmark runner

Benchmark tasks must report workflow-compliance evidence only. They can check sources, artifacts, manifests, and metric availability, but they must not claim paper reproduction success or scientific benchmark scores.

v0.4.0 benchmark tasks may use either legacy expected_artifacts and evaluation_metrics fields or the newer artifacts.required, artifacts.optional, metrics.required, and metrics.optional fields. Optional checks should be reported without failing the benchmark status.

v0.6.0 benchmark tasks may include dataset, environment, dependencies, paper_source, and expected_runtime_notes fields. These fields are provenance evidence only. They must be surfaced in benchmark outputs and indexes without turning them into scientific scores.

Project inspection

openrepro inspect <project_name> should print a concise human-facing table and write workspace/inspect_summary.json. The JSON summary is the stable interface for other agents; update tests whenever the summary schema changes.

Experiment scaffolds

openrepro scaffold-experiment creates files under experiments/<id>/ from candidate evidence. Generated scaffolds must stay human-gated: mark candidates as unverified, write APPROVAL_REQUIRED.md, and avoid runnable scientific claims until a human has reviewed formulas, parameters, and assumptions.

v0.5.0 adds openrepro approve-candidates, which writes workspace/verified_candidates.json and workspace/VERIFIED_CANDIDATES.md. Experiment scaffolds may use that artifact to mark inputs as verified_inputs_ready, but this still does not claim paper reproduction success.

v0.7.1 adds openrepro list-candidates and openrepro review-candidates. Review status values are verified_by_human, rejected_by_human, and needs_more_evidence; unreviewed candidates remain candidate_unverified. Reviews should be written to workspace/candidate_reviews.json and workspace/CANDIDATE_REVIEWS.md. A verified_by_human review should also update the verified candidate artifact so existing experiment guardrails keep working.

openrepro run-experiment may execute only verified experiment scaffolds. It must require --confirm, require status: verified_inputs_ready, capture execution evidence under a normal output run directory, and write a manifest. It must not reinterpret a successful runner exit as scientific reproduction success.

v0.8.1 adds experiment templates. scaffold-experiment --template basic keeps the guarded placeholder behavior, while --template boc-like and --template numeric-sweep generate starter runners for verified scaffolds. Template runners must write outputs under OPENREPRO_RUN_DIR, and run-experiment must include expected_artifacts.json required paths in the run manifest. If a completed runner omits a required template artifact, run-experiment should fail instead of leaving the issue for a later validation step.

v0.8.2 centralizes template metadata in experiment_templates.py. New templates should be added there first, then exercised through openrepro list-templates, scaffold generation, inspect/status summaries, and expected-artifact diagnostics.

v0.9.0 adds experiment_inputs.py. Scaffold generation should map human-verified candidates into experiments/<id>/experiment_inputs.json. Template runners should read the file from OPENREPRO_EXPERIMENT_INPUTS, use candidate-derived parameter_values when present, and fall back to documented defaults when completeness warnings remain. run-experiment should snapshot the input file under configs/experiment_inputs_snapshot.json.

v0.9.1 adds environment_snapshot.py. Experiment runs should record Python, platform, dependency versions, random seed, runner hash, and a lightweight same-seed repeatability check under configs/environment_snapshot.json. Lineage entries for run-experiment should include hashes for experiment config, experiment inputs, environment snapshot, and runner code.

v0.9.2 makes experiment inputs editable and auditable. validate-inputs should refresh completeness and write workspace validation artifacts. set-input should record input sources, preserve manual overrides, and keep generated runner behavior tied to experiment_inputs.json.

v1.2.0 adds experiment_spec.py. Scaffold generation should create experiment_spec.json; run-experiment should validate and snapshot it before execution. Spec validation is an engineering contract check and must not be presented as scientific correctness.

v1.2.1 makes specs freshness-aware. Spec source fingerprints should be stable over config, input, expected artifact, and metric contracts, while ignoring timestamps. validate-experiment-spec --strict should report stale specs without refreshing them, and comparison reports should warn when two experiment runs used different spec hashes.

v1.3.0 adds data_registry.py. Data registration should record local file path, role, size, and SHA-256 under workspace/data_index.json without copying or inventing datasets. validate-data should detect missing files and hash mismatches. Experiment specs should include the registered data contract, and run-experiment should snapshot the data index under configs/data_index_snapshot.json for lineage and evidence packages.

v1.4.0 adds quality_gate.py. run-experiment should write reports/quality_gate.json and reports/quality_gate.md after manifest generation. Quality gate reports are intentionally excluded from run manifests so manual re-evaluation does not stale the manifest; lineage and evidence packages record their hashes separately. Quality gates should check execution evidence completeness: manifest validity, runner completion, required metrics, spec snapshots, data index snapshots, and environment snapshots. They must not be presented as scientific reproduction success.

v1.4.1 extends quality gates into project-level summaries and diagnostics. quality-gate --all should evaluate every run and write workspace/quality_gate_summary.json plus workspace/QUALITY_GATE_SUMMARY.md. Diagnostic issues should include failed gate check names so repair planning can suggest the producing command or artifact class without fabricating evidence.

v1.5.0 makes repair planning quality-gate-aware. Diagnostics should emit both an aggregate quality_gate_failed issue and check-specific issue codes such as quality_gate_metrics_missing, quality_gate_runner_failed, or quality_gate_data_snapshot_missing. Repair plans and dry-runs should map those codes to explicit recovery actions, but they must not fabricate metrics, snapshots, datasets, or scientific outputs.

v1.6.0 adds claim_trace.py. Claim traces should treat formula and parameter candidates as traceable claims, then link them to experiment specs, registered data, and run evidence. trace-claims should write JSON and Markdown under workspace/. The trace is an audit map only and must not imply that a claim was scientifically reproduced.

v1.6.1 adds claim trace validation. validate-claims should check trace freshness and link integrity without rewriting claim_trace.json. Validation may flag stale traces, unresolved experiment claims, unregistered experiment data, and experiment runs that no longer point to known scaffolds. These checks are engineering audit checks only; they must not verify formulas, datasets, or scientific correctness.

v1.7.0 adds scorecard.py. Readiness scorecards should aggregate workflow evidence completeness across paper evidence, candidate review, data provenance, experiment specs, run evidence, quality gates, repeatability evidence, and claim trace health. Scores are for triage and handoff only. They must not be described as scientific reproduction scores or proof that a paper result was reproduced.

v1.7.1 adds gaps.py. Reproduction gaps should convert upstream workflow evidence issues into severity-ranked to-dos with suggested commands. Gaps may use scorecard dimensions, diagnostics, quality gates, and claim trace validation, but should avoid evaluating generated evidence packages directly to prevent self-referential freshness loops. Closing gaps is workflow housekeeping, not a scientific reproduction claim.

v1.8.0 adds checkpoints.py. Workflow checkpoints should normalize major project stages into complete, partial, blocked, or missing, and should point to a single next checkpoint plus command. Checkpoints are allowed to read existing summaries, but they should not generate scientific evidence, run experiments, or imply that checkpoint completion equals paper reproduction.

v1.8.1 adds advance.py. Advance plans should be dry-run previews only. They may select a command from open gaps or the next incomplete checkpoint, but they must not execute commands, fill placeholder values, run experiments, repair artifacts, or generate scientific results. The CLI should require advance --dry-run until an explicitly reviewed apply mode exists.

v1.9.0 adds review_board.py. Review boards should aggregate existing human review prompts into a single queue. They may point to candidate review, data, experiment spec, claim trace, scorecard, gap, or advance-plan commands, but they must not mark formulas, data, code, or scientific outputs as validated. A clear board means no open workflow prompts were detected, not that the paper was reproduced.

v1.9.1 adds review_decisions.py. Review decisions should record human handling of review board items with reviewer notes and explicit status. A closed item means the review prompt was handled or rejected; it must not be treated as formula validation, data validation, code correctness, or scientific reproduction evidence.

v1.10.0 adds reproduction_protocol.py. Protocols should synthesize target claims, required data, required experiments, required runs, and acceptance criteria from existing artifacts. They may say whether workflow criteria are blocked or ready, but they must not claim that the protocol has been executed successfully or that the paper has been scientifically reproduced.

v1.10.1 adds protocol_coverage.py. Coverage should check whether protocol claims, data, experiments, runs, and acceptance criteria are linked to current workflow evidence. Coverage scores are engineering completeness signals only; they must not be framed as scientific reproduction scores.

v1.11.0 adds protocol_plan.py. Protocol plans should convert protocol coverage gaps into prioritized, advisory workflow actions. They may surface a top command and placeholder-input requirements, but they must not execute commands, fill unknown scientific values, or present planned actions as reproduction evidence.

v1.11.1 adds protocol_preflight.py. Preflight should check protocol readiness before execution or handoff by aggregating protocol, coverage, plan, data, spec, quality-gate, and review-decision state. Missing or stale evidence packages may be warnings, but blockers must still point to commands that generate real workflow evidence rather than fabricating results.

v1.12.0 adds claim_evidence_binder.py. Binders should organize each traced claim with linked experiments, runs, data ids, quality gates, protocol coverage, and review decisions. Missing evidence lists should drive workflow follow-up, but a complete binder must not be described as scientific proof that a paper claim was reproduced.

v1.12.1 adds claim evidence binder validation. Validation should compare the stored binder against current claims, coverage, review decisions, data, specs, and runs, then report stale or internally inconsistent binders. It may suggest regenerating the binder, but it must not rewrite binder files before detecting staleness or present validation as scientific correctness.

v1.13.0 adds claim_signoff.py. Claim signoffs should record explicit human decisions for each claim evidence binder record. accepted_workflow_evidence means the reviewer accepts the workflow evidence record as organized, not that the paper claim has been scientifically reproduced. Non-terminal decisions should preserve follow-up commands or notes instead of fabricating missing evidence.

v1.13.1 adds claim_evidence_report.py. Claim evidence reports should merge the binder, binder validation, and latest signoffs into a reviewer-facing matrix. Reports may identify open workflow actions and top commands, but they must keep the distinction between workflow evidence and scientific proof.

v1.14.0 adds claim_signoff_validation.py. Signoff validation should check coverage, freshness, orphan records, and accepted incomplete claims against the current claim evidence binder. It may invalidate workflow signoffs after evidence changes, but it must not alter human decisions or claim scientific correctness.

v1.14.1 adds claim_evidence_report_validation.py. Report validation should compare the stored report against the current binder, binder validation, and claim signoffs without overwriting the report before checking freshness. It may flag stale reports and inconsistent ready states, but it must not convert report readiness into a scientific reproduction claim.

v1.15.0 adds reviewer_packet.py. Reviewer packets should consolidate claim evidence, signoffs, validations, open actions, review order, and source artifact hashes for human handoff. They may export a zip for convenience, but the packet must remain workflow review evidence rather than a claim of scientific reproduction.

v1.16.0 adds review_site.py. Review sites should be static HTML handoff views over existing workflow evidence, with a JSON manifest and optional zip. Exclude generated review-site files from evidence-package freshness fingerprints so the site can be generated after a current evidence package without making that package stale.

v1.16.1 adds timeline.py. Project timelines should aggregate existing workflow evidence and human decisions into chronological JSON/Markdown without claiming scientific validation. Generate timelines before evidence packages so the package fingerprint includes the timeline, and keep review sites as a read-only view over the packaged evidence.

v1.17.0 adds collaboration_pack.py. Collaboration packs should be role-based handoff views for maintainers, reviewers, experimenters, and the next agent. They should list safe commands and files to inspect first, but remain a coordination layer rather than evidence that scientific reproduction succeeded. Generated collaboration pack files are excluded from evidence-package freshness fingerprints because they are downstream handoff artifacts.

v1.18.0 adds refresh.py. Refresh runs should regenerate derived workflow and handoff artifacts only: quality gates for existing runs, lineage, traceability, scorecards, gaps, protocols, binders, reviewer packets, timelines, reports, handoff files, evidence packages, review sites, and collaboration packs. They must not run experiments, add claim signoffs, close review decisions, or fabricate missing scientific artifacts. Generated refresh files are excluded from evidence-package freshness fingerprints because they are orchestration records, not source evidence.

v1.18.1 adds freshness.py. Artifact freshness graphs should explain why derived artifacts are current, stale, or missing. They may compare evidence package fingerprints against current source evidence and list dependency edges among handoff artifacts, but they should remain diagnostic records. Generated freshness graph files are excluded from evidence-package freshness fingerprints.

v1.19.0 adds dashboard.py. Dashboards should be static project handoff views that aggregate readiness, freshness, refresh, collaboration, timeline, reviewer, review-site, evidence-package, and handoff links. Generated dashboard files are excluded from evidence-package freshness fingerprints because they are downstream navigation artifacts, not source evidence.

v1.20.0 adds project_profile.py. Project profiles should define reproduction scope, target claims, required data, required experiments, and acceptance dimensions before reports and evidence packages are finalized. They are source evidence for scope and should remain included in evidence-package fingerprints. Profiles must not claim that scientific reproduction has succeeded.

v1.20.1 adds acceptance_criteria.py. Acceptance criteria should evaluate workflow readiness gates derived from the project profile: reviewed claims, current data, experiment contracts, run evidence, quality gates, trace validation, scorecard gaps, and protocol readiness. They are source evidence for readiness gates and should remain included in evidence-package fingerprints. Acceptance criteria must not claim scientific reproduction success.

v1.21.0 adds readiness_review.py. Readiness reviews should be final human-handoff reports that check downstream artifacts such as evidence package, freshness, refresh, dashboard, review site, reviewer packet, and collaboration pack after source evidence has been finalized. Generated readiness review files are excluded from evidence-package freshness fingerprints because they are downstream reports, not source evidence.

v1.21.1 adds readiness_review_validation.py. Readiness review validation should compare a stored readiness review against the current built review using a stable fingerprint that ignores volatile fields such as timestamps and output paths. Generated readiness review validation files are excluded from evidence-package freshness fingerprints because they validate a downstream report, not source evidence.

v1.22.0 adds review_action_plan.py. Review action plans should turn blocked readiness review checks into role-based advisory tasks with priorities and commands. They must not execute commands, close human decisions, or mutate scientific evidence. Generated review action plan files are excluded from evidence-package freshness fingerprints because they are downstream planning artifacts.

v1.22.1 adds delivery_bundle.py. Delivery bundles should collect existing final workflow handoff files into a JSON/Markdown manifest and optional zip. They must not run experiments, close review decisions, add signoffs, or fabricate missing scientific artifacts. Generated delivery bundle files are excluded from evidence-package freshness fingerprints because they are downstream handoff artifacts.

v1.23.0 adds multi_agent_plan.py. Multi-agent plans should define role ownership and assign open workflow tasks from existing artifacts such as review action plans, collaboration packs, delivery bundles, and project status. They must remain advisory unless a future explicitly supervised execution adapter is added. Generated multi-agent plan files are excluded from evidence-package freshness fingerprints because they are downstream coordination artifacts.

v1.23.1 adds multi_agent_plan_validation.py. Multi-agent plan validation should check required fields, task counts, freshness against current project status, role IDs, priorities, task status, and unsafe commands. It must remain read-only: validation should not regenerate plans, execute agents, run experiments, close human decisions, add signoffs, or fabricate missing scientific artifacts. Generated validation files are excluded from evidence-package freshness fingerprints because they are downstream coordination artifacts.

v1.24.0 adds agent_board.py. Agent boards should render the guarded multi-agent plan as a static HTML/JSON handoff view with one lane per role. They may generate missing upstream plan/validation artifacts, but must not dispatch agents, execute task commands, run experiments, close human decisions, add signoffs, or fabricate scientific artifacts. Generated board files are excluded from evidence-package freshness fingerprints because they are downstream coordination artifacts.

v1.24.1 adds agent_dispatch.py. Dispatch packs should split guarded multi-agent tasks into per-role Markdown task files plus a JSON/Markdown manifest. They may generate missing upstream plan/validation artifacts, but must not launch agents, execute task commands, run experiments, close human decisions, add signoffs, or fabricate scientific artifacts. Generated dispatch files are excluded from evidence-package freshness fingerprints because they are downstream coordination artifacts.

v1.25.0 adds agent_exec_plan.py. Execution plans are dry-run classifiers for dispatch tasks. They may mark safe derived-artifact commands as would-run steps, but must never execute commands. Experiments, reruns, review decisions, claim signoffs, repair apply, human-input tasks, and placeholder commands must remain blocked. Generated execution-plan files are excluded from evidence-package freshness fingerprints because they are downstream coordination artifacts.

v1.26.0 adds paper_lineage.py. Paper lineage graphs should organize existing workflow evidence into claim, method, data, experiment, and metric nodes. They must not infer missing paper content, verify scientific correctness, run experiments, fabricate metrics, or claim reproduction success. Generated paper-lineage files are excluded from evidence-package freshness fingerprints because they are downstream evidence-organization artifacts.

v0.9.3 adds experiment_compare.py. rerun-experiment should reuse the same execution guardrails as run-experiment, while compare-experiments should compare only runs that belong to the requested experiment id. Comparison reports must stay evidence-oriented: metric deltas, runner hashes, raw input hashes, normalized input hashes, and environment hashes are acceptable; scientific reproduction claims are not.

v1.0.0 adds evidence_package.py. Evidence packages should reuse existing inspect, lineage, manifest validation, experiment, benchmark, report, and handoff artifacts rather than recomputing incompatible state. They should write both JSON and Markdown under reports/ and keep policy language explicit that workflow evidence is not a scientific reproduction claim.

v1.0.1 adds evidence_fingerprint.py. Evidence package freshness should be based on source fingerprints that exclude generated evidence package files. evidence-package --zip should export the package plus referenced workspace and handoff artifacts without changing scientific evidence.

Run lineage

openrepro lineage <project_name> writes workspace/run_lineage.json and workspace/RUN_LINEAGE.md. Entries should include parent command, manifest hash, config hash, source index hash, and verified candidate hash when present. Experiment reruns should be grouped with repeat ids and repeat run indexes. Lineage artifacts are provenance evidence only.

openrepro inspect, openrepro status, and handoff files should surface whether lineage has been generated so agents can avoid guessing project provenance state.

Doctor checks

openrepro doctor <project_name> writes workspace/doctor.json and workspace/DOCTOR.md. Doctor checks should stay focused on environment and workflow readiness: dependency availability, project structure, config presence, and provider readiness. They must not infer scientific validity.

Repair and run comparison

openrepro repair-plan is advisory only in v0.4.0 and must not mutate project code or artifacts. openrepro compare-runs should report observed manifest and metric differences without interpreting them as scientific superiority.

v0.5.0 adds openrepro repair --dry-run, which writes workspace/repair_dry_run.json and workspace/REPAIR_DRY_RUN.md without mutating run artifacts. Manifest repair previews should be generated as diffs against the current on-disk files; missing scientific artifacts must never be fabricated.

v0.6.1 adds openrepro repair --apply --only manifest --confirm. Apply mode must remain manifest-only: regenerate manifest.json from files already present on disk, write workspace/repair_apply.json and workspace/REPAIR_APPLY.md, and do not generate scientific artifacts, edit experiment code, or infer parameters.