fix(jobs): id reuse across restarts, same-second reconcile, kill race, trim-temp orphans

[sebyx07]

Bug-hunt findings in the job engine, all with tests:

1. Cross-restart job-id collision (data corrupting). JobId::generate's exists check only consulted the in-memory map, but the id namespace (DB rows + <id>.log files) survives restarts for 24h. A new job minting a retained id would truncate the old job's log, lose its own row to the PK conflict, and persist_final would clobber the old row. Now the on-disk log counts as taken, and suffixed candidates (<base>-1, which also restarts each boot) are re-checked in a loop.
2. Same-second restart left a dead job 'running' for 24h. Reconcile was scoped started_unix < boot; a crash loop restarting within one wall-clock second missed the previous process's row. Now <= boot excluding ids live in this process (snapshotted under the jobs lock, so this process's own jobs are never clobbered). Also fixes the NOT IN () empty-list syntax error.
3. Kill race misreported success as failure. If the group died between the TERM grace expiring and the SIGKILL, kill_job returned false even though its own TERM worked. Now reports by final job state.
4. Orphaned .log.trim temps accumulated forever. A trim crashing between write and rename left a temp no reaper path ever matched (extension trim, not log). reap_orphans now sweeps them past the mtime gate.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Improved job ID generation to keep trying new numeric suffixes until an unused ID is found.
  • Prevented stale running jobs from being marked failed during startup if they belong to the current process.
  • Avoided reusing retained job IDs after restart, helping preserve existing job output and final state.
  • Made job termination reporting more accurate when a job exits during the shutdown window.
  • Expanded cleanup to remove leftover temporary trim files while keeping valid job logs.

[coderabbitai]

Walkthrough

Job id generation now loops through suffix candidates re-checking availability instead of appending one fixed suffix. JobStore's restart reconcile excludes live in-memory job ids and widens staleness comparison, and rejects id reuse when a retained log exists. Reaper now sweeps aged .trim temp files and rechecks final job state after failed kill delivery. Tests added for each behavior.

Changes

Job lifecycle correctness fixes

Layer / File(s)	Summary
JobId collision retry loop src/jobs/id.rs	generate loops incrementing sequence and re-checking each base-n candidate against exists until a free one is found; tests updated/added for single and multi-suffix collisions.
Restart-safe reconcile and id reuse src/jobs/mod.rs	Startup reconcile excludes current in-memory job ids and marks started_unix <= boot running rows as failed; run rejects ids present in-memory or with a retained .log file; tests cover same-second reconcile and post-restart id non-reuse.
Reaper kill outcome and trim sweeping src/jobs/reaper.rs	kill_job returns based on rechecked final job state instead of assuming failure after failed KILL delivery; reap_orphans now also sweeps aged .trim temp files alongside orphaned .log files; new test validates deletion/retention behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Process
  participant JobStore
  participant Database
  participant Filesystem

  Process->>JobStore: new(boot)
  JobStore->>Database: snapshot in-memory job ids
  JobStore->>Database: mark stale running (started_unix <= boot, id NOT IN ids) as failed

  Process->>JobStore: run(job)
  JobStore->>JobStore: JobId::generate loop
  JobStore->>Filesystem: check {id}.log exists
  Filesystem-->>JobStore: exists/not exists
  JobStore-->>Process: unique job id assigned

Loading

Estimated code review effort: 3 (Moderate) | ~25 minutes

Related PRs: None specified.

Suggested labels: bug, jobs, reliability

Suggested reviewers: None specified.

Poem:
A rabbit hops through logs and boot,
Checking suffixes root by root,
No stale job stays, no id reused,
No trim file left to lie confused,
Kill or not, the state's the truth —
Thump thump, job's status: proof! 🐇

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main jobs-related fixes: restart ID reuse, same-second reconcile, kill race handling, and trim-temp orphan cleanup.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/job-engine-correctness

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/jobs/reaper.rs`:
- Around line 462-472: The test setup in reaper’s retention-age block currently
uses the external touch command with the GNU-only -d `@0` flag, which is not
portable. Update the logic in the loop over trim_tmp, orphan, and known to set
the file’s modified time in-process using std::fs::File::set_modified with
UNIX_EPOCH instead of spawning touch, so the age-reset step works on BSD/macOS
as well.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: adcf214d-56f8-4315-837f-4de5b6a0c7b7

📥 Commits

Reviewing files that changed from the base of the PR and between d8b41a0 and 13e0e79.

📒 Files selected for processing (3)

• src/jobs/id.rs
• src/jobs/mod.rs
• src/jobs/reaper.rs