pip install -e ".[dev]"
python -m pytest -qOn Windows, if pytest cannot access the default temp directory, use a workspace-local base temp directory:
New-Item -ItemType Directory -Force .codex_tmp\pytest-basetemp | Out-Null
python -m pytest -q --basetemp .codex_tmp\pytest-basetempbash scripts/smoke_test.shPowerShell:
.\scripts\smoke_test.ps1- Add implementation in a focused module under
src/openrepro/. - Add a Typer command in
cli.py. - Add tests under
tests/. - Update README and docs.
- Ensure generated artifacts are timestamped or safely overwritten only when appropriate.
Demo and sweep runs must write manifest.json after all run artifacts are complete.
When adding a new run command, define its required artifacts in artifact_manager.py,
write the manifest at the end of the command, and add validation tests.
PDF ingestion uses pdfplumber. Extracted text and page provenance should stay under
workspace/extracted_sources/, and all generated formulas or parameters must remain
marked as candidates until a human verifies them.
v0.8.0 candidate extraction should preserve evidence provenance: source path, chunk index, page number when available, context windows, and evidence quality signals. These fields help review candidates; they are not verification results.
v1.1.0 candidate extraction should also preserve section labels, caption anchors, and risk flags. Risk flags are triage hints for human review, not validation results or automated rejection decisions.
Real providers should be opt-in. They must write actual usage records and must not fabricate tokens or costs.
v0.4.0 ships MockProvider plus a minimal OpenAI-compatible provider path.
Real calls must require explicit configuration, environment-backed secrets, and
request-hash cache accounting. Do not store API keys in project files.
v0.5.2 adds provider cache policy and preview redaction. New provider code should
write redacted prompt_preview and response_preview fields to usage records,
store cache files under provider/model/task namespaces, and respect
cache_enabled, cache_ttl_seconds, and redact_prompts.
Benchmark tasks must report workflow-compliance evidence only. They can check sources, artifacts, manifests, and metric availability, but they must not claim paper reproduction success or scientific benchmark scores.
v0.4.0 benchmark tasks may use either legacy expected_artifacts and
evaluation_metrics fields or the newer artifacts.required,
artifacts.optional, metrics.required, and metrics.optional fields.
Optional checks should be reported without failing the benchmark status.
v0.6.0 benchmark tasks may include dataset, environment, dependencies, paper_source, and expected_runtime_notes fields. These fields are provenance evidence only. They must be surfaced in benchmark outputs and indexes without turning them into scientific scores.
openrepro inspect <project_name> should print a concise human-facing table and
write workspace/inspect_summary.json. The JSON summary is the stable interface
for other agents; update tests whenever the summary schema changes.
openrepro scaffold-experiment creates files under experiments/<id>/ from
candidate evidence. Generated scaffolds must stay human-gated: mark candidates
as unverified, write APPROVAL_REQUIRED.md, and avoid runnable scientific claims
until a human has reviewed formulas, parameters, and assumptions.
v0.5.0 adds openrepro approve-candidates, which writes
workspace/verified_candidates.json and workspace/VERIFIED_CANDIDATES.md.
Experiment scaffolds may use that artifact to mark inputs as
verified_inputs_ready, but this still does not claim paper reproduction
success.
v0.7.1 adds openrepro list-candidates and openrepro review-candidates.
Review status values are verified_by_human, rejected_by_human, and
needs_more_evidence; unreviewed candidates remain candidate_unverified.
Reviews should be written to workspace/candidate_reviews.json and
workspace/CANDIDATE_REVIEWS.md. A verified_by_human review should also
update the verified candidate artifact so existing experiment guardrails keep
working.
openrepro run-experiment may execute only verified experiment scaffolds. It
must require --confirm, require status: verified_inputs_ready, capture
execution evidence under a normal output run directory, and write a manifest.
It must not reinterpret a successful runner exit as scientific reproduction
success.
v0.8.1 adds experiment templates. scaffold-experiment --template basic keeps
the guarded placeholder behavior, while --template boc-like and
--template numeric-sweep generate starter runners for verified scaffolds.
Template runners must write outputs under OPENREPRO_RUN_DIR, and
run-experiment must include expected_artifacts.json required paths in the
run manifest. If a completed runner omits a required template artifact,
run-experiment should fail instead of leaving the issue for a later
validation step.
v0.8.2 centralizes template metadata in experiment_templates.py.
New templates should be added there first, then exercised through
openrepro list-templates, scaffold generation, inspect/status summaries, and
expected-artifact diagnostics.
v0.9.0 adds experiment_inputs.py. Scaffold generation should map
human-verified candidates into experiments/<id>/experiment_inputs.json.
Template runners should read the file from OPENREPRO_EXPERIMENT_INPUTS, use
candidate-derived parameter_values when present, and fall back to documented
defaults when completeness warnings remain. run-experiment should snapshot
the input file under configs/experiment_inputs_snapshot.json.
v0.9.1 adds environment_snapshot.py. Experiment runs should record Python,
platform, dependency versions, random seed, runner hash, and a lightweight
same-seed repeatability check under configs/environment_snapshot.json.
Lineage entries for run-experiment should include hashes for experiment
config, experiment inputs, environment snapshot, and runner code.
v0.9.2 makes experiment inputs editable and auditable. validate-inputs should
refresh completeness and write workspace validation artifacts. set-input
should record input sources, preserve manual overrides, and keep generated
runner behavior tied to experiment_inputs.json.
v1.2.0 adds experiment_spec.py. Scaffold generation should create
experiment_spec.json; run-experiment should validate and snapshot it before
execution. Spec validation is an engineering contract check and must not be
presented as scientific correctness.
v1.2.1 makes specs freshness-aware. Spec source fingerprints should be stable
over config, input, expected artifact, and metric contracts, while ignoring
timestamps. validate-experiment-spec --strict should report stale specs
without refreshing them, and comparison reports should warn when two experiment
runs used different spec hashes.
v1.3.0 adds data_registry.py. Data registration should record local file path,
role, size, and SHA-256 under workspace/data_index.json without copying or
inventing datasets. validate-data should detect missing files and hash
mismatches. Experiment specs should include the registered data contract, and
run-experiment should snapshot the data index under
configs/data_index_snapshot.json for lineage and evidence packages.
v1.4.0 adds quality_gate.py. run-experiment should write
reports/quality_gate.json and reports/quality_gate.md after manifest
generation. Quality gate reports are intentionally excluded from run manifests
so manual re-evaluation does not stale the manifest; lineage and evidence
packages record their hashes separately. Quality gates should check execution
evidence completeness: manifest validity, runner completion, required metrics,
spec snapshots, data index snapshots, and environment snapshots. They must not
be presented as scientific reproduction success.
v1.4.1 extends quality gates into project-level summaries and diagnostics.
quality-gate --all should evaluate every run and write
workspace/quality_gate_summary.json plus workspace/QUALITY_GATE_SUMMARY.md.
Diagnostic issues should include failed gate check names so repair planning can
suggest the producing command or artifact class without fabricating evidence.
v1.5.0 makes repair planning quality-gate-aware. Diagnostics should emit both
an aggregate quality_gate_failed issue and check-specific issue codes such as
quality_gate_metrics_missing, quality_gate_runner_failed, or
quality_gate_data_snapshot_missing. Repair plans and dry-runs should map those
codes to explicit recovery actions, but they must not fabricate metrics,
snapshots, datasets, or scientific outputs.
v1.6.0 adds claim_trace.py. Claim traces should treat formula and parameter
candidates as traceable claims, then link them to experiment specs, registered
data, and run evidence. trace-claims should write JSON and Markdown under
workspace/. The trace is an audit map only and must not imply that a claim was
scientifically reproduced.
v1.6.1 adds claim trace validation. validate-claims should check trace
freshness and link integrity without rewriting claim_trace.json. Validation
may flag stale traces, unresolved experiment claims, unregistered experiment
data, and experiment runs that no longer point to known scaffolds. These checks
are engineering audit checks only; they must not verify formulas, datasets, or
scientific correctness.
v1.7.0 adds scorecard.py. Readiness scorecards should aggregate workflow
evidence completeness across paper evidence, candidate review, data provenance,
experiment specs, run evidence, quality gates, repeatability evidence, and claim
trace health. Scores are for triage and handoff only. They must not be described
as scientific reproduction scores or proof that a paper result was reproduced.
v1.7.1 adds gaps.py. Reproduction gaps should convert upstream workflow
evidence issues into severity-ranked to-dos with suggested commands. Gaps may
use scorecard dimensions, diagnostics, quality gates, and claim trace validation,
but should avoid evaluating generated evidence packages directly to prevent
self-referential freshness loops. Closing gaps is workflow housekeeping, not a
scientific reproduction claim.
v1.8.0 adds checkpoints.py. Workflow checkpoints should normalize major
project stages into complete, partial, blocked, or missing, and should
point to a single next checkpoint plus command. Checkpoints are allowed to read
existing summaries, but they should not generate scientific evidence, run
experiments, or imply that checkpoint completion equals paper reproduction.
v1.8.1 adds advance.py. Advance plans should be dry-run previews only. They
may select a command from open gaps or the next incomplete checkpoint, but they
must not execute commands, fill placeholder values, run experiments, repair
artifacts, or generate scientific results. The CLI should require
advance --dry-run until an explicitly reviewed apply mode exists.
v1.9.0 adds review_board.py. Review boards should aggregate existing human
review prompts into a single queue. They may point to candidate review, data,
experiment spec, claim trace, scorecard, gap, or advance-plan commands, but they
must not mark formulas, data, code, or scientific outputs as validated. A clear
board means no open workflow prompts were detected, not that the paper was
reproduced.
v1.9.1 adds review_decisions.py. Review decisions should record human
handling of review board items with reviewer notes and explicit status. A
closed item means the review prompt was handled or rejected; it must not be
treated as formula validation, data validation, code correctness, or scientific
reproduction evidence.
v1.10.0 adds reproduction_protocol.py. Protocols should synthesize target
claims, required data, required experiments, required runs, and acceptance
criteria from existing artifacts. They may say whether workflow criteria are
blocked or ready, but they must not claim that the protocol has been executed
successfully or that the paper has been scientifically reproduced.
v1.10.1 adds protocol_coverage.py. Coverage should check whether protocol
claims, data, experiments, runs, and acceptance criteria are linked to current
workflow evidence. Coverage scores are engineering completeness signals only;
they must not be framed as scientific reproduction scores.
v1.11.0 adds protocol_plan.py. Protocol plans should convert protocol
coverage gaps into prioritized, advisory workflow actions. They may surface a
top command and placeholder-input requirements, but they must not execute
commands, fill unknown scientific values, or present planned actions as
reproduction evidence.
v1.11.1 adds protocol_preflight.py. Preflight should check protocol readiness
before execution or handoff by aggregating protocol, coverage, plan, data,
spec, quality-gate, and review-decision state. Missing or stale evidence
packages may be warnings, but blockers must still point to commands that
generate real workflow evidence rather than fabricating results.
v1.12.0 adds claim_evidence_binder.py. Binders should organize each traced
claim with linked experiments, runs, data ids, quality gates, protocol coverage,
and review decisions. Missing evidence lists should drive workflow follow-up,
but a complete binder must not be described as scientific proof that a paper
claim was reproduced.
v1.12.1 adds claim evidence binder validation. Validation should compare the stored binder against current claims, coverage, review decisions, data, specs, and runs, then report stale or internally inconsistent binders. It may suggest regenerating the binder, but it must not rewrite binder files before detecting staleness or present validation as scientific correctness.
v1.13.0 adds claim_signoff.py. Claim signoffs should record explicit human
decisions for each claim evidence binder record. accepted_workflow_evidence
means the reviewer accepts the workflow evidence record as organized, not that
the paper claim has been scientifically reproduced. Non-terminal decisions
should preserve follow-up commands or notes instead of fabricating missing
evidence.
v1.13.1 adds claim_evidence_report.py. Claim evidence reports should merge
the binder, binder validation, and latest signoffs into a reviewer-facing
matrix. Reports may identify open workflow actions and top commands, but they
must keep the distinction between workflow evidence and scientific proof.
v1.14.0 adds claim_signoff_validation.py. Signoff validation should check
coverage, freshness, orphan records, and accepted incomplete claims against the
current claim evidence binder. It may invalidate workflow signoffs after
evidence changes, but it must not alter human decisions or claim scientific
correctness.
v1.14.1 adds claim_evidence_report_validation.py. Report validation should
compare the stored report against the current binder, binder validation, and
claim signoffs without overwriting the report before checking freshness. It may
flag stale reports and inconsistent ready states, but it must not convert report
readiness into a scientific reproduction claim.
v1.15.0 adds reviewer_packet.py. Reviewer packets should consolidate claim
evidence, signoffs, validations, open actions, review order, and source artifact
hashes for human handoff. They may export a zip for convenience, but the packet
must remain workflow review evidence rather than a claim of scientific
reproduction.
v1.16.0 adds review_site.py. Review sites should be static HTML handoff views
over existing workflow evidence, with a JSON manifest and optional zip. Exclude
generated review-site files from evidence-package freshness fingerprints so the
site can be generated after a current evidence package without making that
package stale.
v1.16.1 adds timeline.py. Project timelines should aggregate existing
workflow evidence and human decisions into chronological JSON/Markdown without
claiming scientific validation. Generate timelines before evidence packages so
the package fingerprint includes the timeline, and keep review sites as a
read-only view over the packaged evidence.
v1.17.0 adds collaboration_pack.py. Collaboration packs should be role-based
handoff views for maintainers, reviewers, experimenters, and the next agent.
They should list safe commands and files to inspect first, but remain a
coordination layer rather than evidence that scientific reproduction succeeded.
Generated collaboration pack files are excluded from evidence-package freshness
fingerprints because they are downstream handoff artifacts.
v1.18.0 adds refresh.py. Refresh runs should regenerate derived workflow and
handoff artifacts only: quality gates for existing runs, lineage, traceability,
scorecards, gaps, protocols, binders, reviewer packets, timelines, reports,
handoff files, evidence packages, review sites, and collaboration packs. They
must not run experiments, add claim signoffs, close review decisions, or
fabricate missing scientific artifacts. Generated refresh files are excluded
from evidence-package freshness fingerprints because they are orchestration
records, not source evidence.
v1.18.1 adds freshness.py. Artifact freshness graphs should explain why
derived artifacts are current, stale, or missing. They may compare evidence
package fingerprints against current source evidence and list dependency edges
among handoff artifacts, but they should remain diagnostic records. Generated
freshness graph files are excluded from evidence-package freshness fingerprints.
v1.19.0 adds dashboard.py. Dashboards should be static project handoff views
that aggregate readiness, freshness, refresh, collaboration, timeline, reviewer,
review-site, evidence-package, and handoff links. Generated dashboard files are
excluded from evidence-package freshness fingerprints because they are
downstream navigation artifacts, not source evidence.
v1.20.0 adds project_profile.py. Project profiles should define reproduction
scope, target claims, required data, required experiments, and acceptance
dimensions before reports and evidence packages are finalized. They are source
evidence for scope and should remain included in evidence-package fingerprints.
Profiles must not claim that scientific reproduction has succeeded.
v1.20.1 adds acceptance_criteria.py. Acceptance criteria should evaluate
workflow readiness gates derived from the project profile: reviewed claims,
current data, experiment contracts, run evidence, quality gates, trace
validation, scorecard gaps, and protocol readiness. They are source evidence
for readiness gates and should remain included in evidence-package
fingerprints. Acceptance criteria must not claim scientific reproduction
success.
v1.21.0 adds readiness_review.py. Readiness reviews should be final
human-handoff reports that check downstream artifacts such as evidence package,
freshness, refresh, dashboard, review site, reviewer packet, and collaboration
pack after source evidence has been finalized. Generated readiness review files
are excluded from evidence-package freshness fingerprints because they are
downstream reports, not source evidence.
v1.21.1 adds readiness_review_validation.py. Readiness review validation
should compare a stored readiness review against the current built review using
a stable fingerprint that ignores volatile fields such as timestamps and output
paths. Generated readiness review validation files are excluded from
evidence-package freshness fingerprints because they validate a downstream
report, not source evidence.
v1.22.0 adds review_action_plan.py. Review action plans should turn blocked
readiness review checks into role-based advisory tasks with priorities and
commands. They must not execute commands, close human decisions, or mutate
scientific evidence. Generated review action plan files are excluded from
evidence-package freshness fingerprints because they are downstream planning
artifacts.
v1.22.1 adds delivery_bundle.py. Delivery bundles should collect existing
final workflow handoff files into a JSON/Markdown manifest and optional zip.
They must not run experiments, close review decisions, add signoffs, or
fabricate missing scientific artifacts. Generated delivery bundle files are
excluded from evidence-package freshness fingerprints because they are
downstream handoff artifacts.
v1.23.0 adds multi_agent_plan.py. Multi-agent plans should define role
ownership and assign open workflow tasks from existing artifacts such as review
action plans, collaboration packs, delivery bundles, and project status. They
must remain advisory unless a future explicitly supervised execution adapter is
added. Generated multi-agent plan files are excluded from evidence-package
freshness fingerprints because they are downstream coordination artifacts.
v1.23.1 adds multi_agent_plan_validation.py. Multi-agent plan validation
should check required fields, task counts, freshness against current project
status, role IDs, priorities, task status, and unsafe commands. It must remain
read-only: validation should not regenerate plans, execute agents, run
experiments, close human decisions, add signoffs, or fabricate missing
scientific artifacts. Generated validation files are excluded from
evidence-package freshness fingerprints because they are downstream
coordination artifacts.
v1.24.0 adds agent_board.py. Agent boards should render the guarded
multi-agent plan as a static HTML/JSON handoff view with one lane per role.
They may generate missing upstream plan/validation artifacts, but must not
dispatch agents, execute task commands, run experiments, close human
decisions, add signoffs, or fabricate scientific artifacts. Generated board
files are excluded from evidence-package freshness fingerprints because they
are downstream coordination artifacts.
v1.24.1 adds agent_dispatch.py. Dispatch packs should split guarded
multi-agent tasks into per-role Markdown task files plus a JSON/Markdown
manifest. They may generate missing upstream plan/validation artifacts, but
must not launch agents, execute task commands, run experiments, close human
decisions, add signoffs, or fabricate scientific artifacts. Generated dispatch
files are excluded from evidence-package freshness fingerprints because they
are downstream coordination artifacts.
v1.25.0 adds agent_exec_plan.py. Execution plans are dry-run classifiers for
dispatch tasks. They may mark safe derived-artifact commands as would-run
steps, but must never execute commands. Experiments, reruns, review decisions,
claim signoffs, repair apply, human-input tasks, and placeholder commands must
remain blocked. Generated execution-plan files are excluded from
evidence-package freshness fingerprints because they are downstream
coordination artifacts.
v1.26.0 adds paper_lineage.py. Paper lineage graphs should organize existing
workflow evidence into claim, method, data, experiment, and metric nodes. They
must not infer missing paper content, verify scientific correctness, run
experiments, fabricate metrics, or claim reproduction success. Generated
paper-lineage files are excluded from evidence-package freshness fingerprints
because they are downstream evidence-organization artifacts.
v0.9.3 adds experiment_compare.py. rerun-experiment should reuse the same
execution guardrails as run-experiment, while compare-experiments should
compare only runs that belong to the requested experiment id. Comparison reports
must stay evidence-oriented: metric deltas, runner hashes, raw input hashes,
normalized input hashes, and environment hashes are acceptable; scientific
reproduction claims are not.
v1.0.0 adds evidence_package.py. Evidence packages should reuse existing
inspect, lineage, manifest validation, experiment, benchmark, report, and
handoff artifacts rather than recomputing incompatible state. They should write
both JSON and Markdown under reports/ and keep policy language explicit that
workflow evidence is not a scientific reproduction claim.
v1.0.1 adds evidence_fingerprint.py. Evidence package freshness should be
based on source fingerprints that exclude generated evidence package files.
evidence-package --zip should export the package plus referenced workspace
and handoff artifacts without changing scientific evidence.
openrepro lineage <project_name> writes workspace/run_lineage.json and
workspace/RUN_LINEAGE.md. Entries should include parent command, manifest
hash, config hash, source index hash, and verified candidate hash when present.
Experiment reruns should be grouped with repeat ids and repeat run indexes.
Lineage artifacts are provenance evidence only.
openrepro inspect, openrepro status, and handoff files should surface whether
lineage has been generated so agents can avoid guessing project provenance
state.
openrepro doctor <project_name> writes workspace/doctor.json and
workspace/DOCTOR.md. Doctor checks should stay focused on environment and
workflow readiness: dependency availability, project structure, config presence,
and provider readiness. They must not infer scientific validity.
openrepro repair-plan is advisory only in v0.4.0 and must not mutate project
code or artifacts. openrepro compare-runs should report observed manifest and
metric differences without interpreting them as scientific superiority.
v0.5.0 adds openrepro repair --dry-run, which writes
workspace/repair_dry_run.json and workspace/REPAIR_DRY_RUN.md without
mutating run artifacts. Manifest repair previews should be generated as diffs
against the current on-disk files; missing scientific artifacts must never be
fabricated.
v0.6.1 adds openrepro repair --apply --only manifest --confirm. Apply mode
must remain manifest-only: regenerate manifest.json from files already present
on disk, write workspace/repair_apply.json and workspace/REPAIR_APPLY.md,
and do not generate scientific artifacts, edit experiment code, or infer
parameters.