fix(runtime): harden durable run lifecycle

[AnsonDev42]

Summary

• make terminal result consumption replayable and isolate retry domains
• add cooperative ingestion, durable ingestion markers, and explicit indeterminate submission resolution
• enforce cross-run OpenAI capacity scopes and release version 0.0.7

Proof

Source SHA: e642d0d01f5424880ad9c4234c8a63c6d9ae8e4c

• uv run pytest -q — passed, 89.35% coverage
• uv run ty check src — passed
• uv run ruff check src tests — passed
• uv run mkdocs build --strict — passed
• uv build — passed; wheel and sdist report 0.0.7
• uv lock --check — passed
• git diff --check — passed

Criterion evidence is provided by focused runtime, SQLite migration, storage-contract, and fake-provider integration tests committed with this SHA. The release design record maps all ten findings to their invariant, boundary, and regression test.

Local live-provider smoke was skipped because no acceptance criterion requires paid provider execution; deterministic fake-provider and durable-storage tests exercise the changed orchestration paths.

Release boundaries

• Already-running arbitrary callbacks and provider SDK calls cannot be forcibly interrupted; configured provider timeouts remain required.
• Non-checkpointed pause resumes only while the process-retained iterator exists.
• Providers without idempotency/discovery use explicit operator resolution for ambiguous create outcomes.
• Legacy partial non-checkpointed runs remain incomplete and require cancel/abandon instead of false completion.

[AnsonDev42]

Proof

Tested source SHA: e642d0d01f5424880ad9c4234c8a63c6d9ae8e4c (base: 2b17164ea72a8169d7091d4486472890038cf983). The local checkout matched this SHA before and after evidence collection; no working-tree changes were made.

Focused deterministic evidence on the tested SHA:

1. Terminal result consumption replays after download failure and after a partial item mutation: test_terminal_batch_is_replayed_after_download_failure_across_sqlite_reopen, test_terminal_replay_skips_already_consumed_rows_after_partial_item_mutation.
2. Submission backoff does not suppress active-batch reconciliation: test_submission_backoff_does_not_suppress_ingestion_reconciliation.
3. Slow ingestion flushes a partial cooperative slice before the historic chunk limit: test_ingestion_time_budget_persists_a_partial_slice_before_old_chunk_limit.
4. Incomplete ingestion prevents terminal status, including empty and partial legacy SQLite upgrade paths: test_storage_contract_incomplete_ingestion_prevents_terminal_status, test_sqlite_upgrade_backfills_empty_legacy_run_as_ingestion_incomplete, test_sqlite_upgrade_backfills_partial_legacy_run_as_ingestion_incomplete.
5. Submission intent survives ambiguous creation and supports both operator-confirmed created and not-created resolution: test_submit_pending_items_preserves_indeterminate_create_intent, test_storage_contract_attaches_operator_confirmed_indeterminate_batch, test_storage_contract_not_created_resolution_requeues_claimed_items.
6. Manual pause stops a non-checkpointed source and same-process resume is exactly-once: test_manual_pause_stops_noncheckpointed_source_and_same_process_resume_is_exactly_once.
7. Shared SQLite capacity reservation is atomic, and legacy submitted work remains included: test_sqlite_submission_capacity_reservation_is_atomic_across_store_instances, test_storage_contract_capacity_includes_legacy_submitted_items.
8. A completed batch cannot clear an unrelated retry delay: test_completed_batch_does_not_clear_existing_retry_backoff.
9. Run.wait(timeout) caps its idle sleep to the remaining deadline. A controlled active batch with poll_interval=10 raised TimeoutError in 0.035 s for a 0.03 s timeout. The documented boundary remains that an already-running arbitrary callback or provider SDK call is not forcibly interrupted.
10. Cancellation clears retry backoff after the final active batch drains: test_cancellation_clears_retry_backoff_after_final_batch_drains.

Command result: the focused pytest selection above completed successfully with 19 passed. The deadline check completed successfully. git diff --check was clean before collection.

No credentials, local filesystem locations, temporary identifiers, or payload data are included here. GitHub Actions is the retained CI evidence owner; the current head has successful lint, build, documentation, multi-version test, secret-scan, and PostgreSQL contract checks: https://github.com/AnsonDev42/batchor/actions/runs/29662233537

No acceptance criterion remains unproven within the documented boundaries.

[AnsonDev42]

Proof refresh

The prior proof on e642d0d01f5424880ad9c4234c8a63c6d9ae8e4c is invalidated by the repair push. This refresh is bound to source SHA 757e9cda7faeaff25f5aa3c8182d22fcf877285d (base 2b17164ea72a8169d7091d4486472890038cf983). The checkout matched the new SHA before and after collection, remained clean, and git diff --check passed.

Focused command: uv run pytest -q --no-cov with the 15 previously documented criterion selectors plus:

• test_wait_deadline_reaches_resume_boundary_polling
• test_storage_contract_stale_terminal_failure_replay_is_idempotent

Result: 22 passed. The storage-contract selectors execute against all configured adapters, so the result includes the cross-adapter stale-replay invariant.

Criterion evidence on the refreshed SHA:

1. Terminal consumption remains replayable after download and partial mutation; the new stale-terminal-failure regression additionally proves a second stale consumer cannot double-count or fail after another consumer commits.
2. Submission backoff does not suppress active reconciliation.
3. Slow ingestion flushes cooperative partial slices.
4. Generic incomplete-ingestion state and empty/partial legacy SQLite migration prevent false terminal status.
5. Ambiguous submission intent survives and both operator resolution paths preserve scoped item state.
6. Manual pause/resume for an in-process non-checkpointed iterator remains exactly-once.
7. Shared SQLite capacity reservation remains atomic and includes legacy submitted work.
8. Successful completion does not clear unrelated retry backoff.
9. The new resume-boundary regression proves an expired Run.wait deadline reaches ingestion polling and prevents a provider poll from starting. The existing controlled active-batch harness was also rerun: wait(timeout=0.03, poll_interval=10) raised TimeoutError in 0.035 s. The documented boundary remains that Batchor cannot forcibly interrupt an arbitrary callback or SDK call already running.
10. Cancellation clears retry backoff once the final active batch drains.

Sanitization: no credentials, machine-local paths, temporary labels, payload data, or personal data are included. GitHub Actions owns retained CI evidence for this SHA: https://github.com/AnsonDev42/batchor/actions/runs/29662537134

CI disclosure: the retained run above is green for secret scan, lint, Python 3.12/3.13/3.14 tests, build, docs, and PostgreSQL contracts. A separate duplicate run on the same SHA contains one Python 3.12 failure in test_start_with_same_run_id_resumes_incomplete_composite_ingestion (next_item_index reached a valid cooperative partial slice before the former fixed 1,000-item boundary); another Python 3.12 run on the identical SHA passed. This is a ship-surface CI discrepancy to resolve or classify, not a gap in the 10 criterion proofs above.

[AnsonDev42]

Final proof

The proof snapshots on e642d0d01f5424880ad9c4234c8a63c6d9ae8e4c and 757e9cda7faeaff25f5aa3c8182d22fcf877285d are invalidated by subsequent repairs. This final proof is bound to source SHA 55aac3af5fd5c1d9cc00f6105bf324bfceff0362 (base 2b17164ea72a8169d7091d4486472890038cf983). The checkout matched this SHA before and after collection, remained clean, and git diff --check passed.

Focused command: uv run pytest -q --no-cov with the complete prior criterion selectors plus:

• test_resume_existing_run_stops_after_active_poll_reaches_deadline
• test_resume_existing_run_skips_direct_submit_after_deadline_expiring_poll
• test_wait_deadline_reaches_resume_boundary_polling
• test_storage_contract_stale_terminal_failure_replay_is_idempotent
• test_start_with_same_run_id_resumes_incomplete_composite_ingestion

Result: 25 passed. The storage contract selectors execute against all configured adapters.

Final criterion map:

1. Terminal consumption replays after download failure and partial mutation; stale concurrent terminal-failure replay is idempotent and cannot double-count an attempt.
2. Submission backoff does not suppress active-batch reconciliation.
3. Cooperative ingestion persists valid partial slices before the former 1,000-item boundary.
4. Generic incomplete-ingestion state plus empty and partial legacy SQLite migration prevent false terminal status.
5. Ambiguous submission intent survives; operator-confirmed created and not-created resolutions preserve scoped item state.
6. Manual pause stops a non-checkpointed iterator and same-process resume remains exactly-once.
7. Shared SQLite capacity reservation is atomic and active legacy submitted work remains counted.
8. Successful completion cannot clear unrelated retry backoff.
9. Deadline evidence covers all repaired seams: expiry before resume-boundary polling prevents a provider call; a poll that reaches the deadline prevents subsequent source work and submission; the checkpoint-complete direct-submit path also stops after an overrun. The independent controlled-batch harness was rerun and wait(timeout=0.03, poll_interval=10) raised TimeoutError in 0.033 s. The documented non-preemptive boundary for an arbitrary callback or SDK call already running is unchanged.
10. Cancellation clears retry backoff after the final active batch drains.

Composite restart recovery now proves the durable invariant directly: any valid cooperative partial checkpoint maps to the exact unconsumed source suffix, persisted indexes are contiguous, restart completes all 1,002 records exactly once, and lineage is preserved. This resolves the prior Python 3.12 timing discrepancy caused by asserting a fixed 1,000-item checkpoint.

Sanitization: no credentials, local filesystem locations, temporary identifiers, payloads, or personal data are included. GitHub Actions owns the durable CI evidence. Both runs on this frozen SHA are green for secret scan, lint, Python 3.12/3.13/3.14 tests, build, strict docs, and PostgreSQL contracts:

• https://github.com/AnsonDev42/batchor/actions/runs/29662747288
• https://github.com/AnsonDev42/batchor/actions/runs/29662746208

No criterion or CI gap remains on the tested SHA.