Skip to content

fix(jobs): true lease token, panic containment, tenant-shard job processing#184

Merged
Ghost-Frame merged 1 commit into
mainfrom
fix/jobs-correctness
Jul 18, 2026
Merged

fix(jobs): true lease token, panic containment, tenant-shard job processing#184
Ghost-Frame merged 1 commit into
mainfrom
fix/jobs-correctness

Conversation

@Ghost-Frame

Copy link
Copy Markdown
Owner

Summary

Lease token integrity. claim_next_job populated Job.claimed_at from a second clock reading in Rust instead of the value the UPDATE stored. When the two reads straddled a second boundary, every lease-gated finalizer (JOB-2) silently no-oped and the row hung at 'running' until recover_stuck_jobs. The UPDATE now uses RETURNING and the struct carries the stored value.

Panic containment. A panicking handler future unwound through process_next_job, killing the whole worker loop, and since no finalizer ran, the row stayed 'running'. A deterministically panicking payload looped forever: the attempts >= max_attempts check lives in exactly the code the panic skipped. The handler future is now wrapped in catch_unwind — still directly owned by tokio::time::timeout, preserving cancel-on-timeout — and a panic feeds the normal retry/fail branching with the reason recorded.

Tenant-shard job processing. In sharded mode, ingestion enqueues jobs into the tenant shard the request resolved, but the worker only polled the monolith: shard jobs sat pending forever. The worker now drains resident active shards every tick (snapshot_all_handles — no loads, no touches, so polling cannot pin tenants resident or fight eviction) and sequentially sweeps all active tenants on the 5-minute stuck-recovery cadence to catch shards evicted with a backlog. The ingestion.fact_extract/ingestion.entity_extract handlers resolve their database from the payload's user_id at execution time instead of closing over the monolith — per-file autoincrement memory ids collide across shards, so the fixed-db lookup could derive facts against another tenant's memory row.

Testing

  • claimed_at_in_struct_matches_stored_row
  • panicking_handler_is_retried_not_stuck / panicking_handler_fails_after_max_attempts (panic reason recorded on the row)
  • worker_drains_resident_tenant_shard_jobs (end-to-end: real TenantRegistry, job enqueued into a shard, worker drains it)
  • cargo test -p kleos-lib -p kleos-server --lib jobs: 19 + 1 passed.

…essing

Three fixes to the durable jobs pipeline:

Lease token integrity. claim_next_job populated Job.claimed_at from a
second clock reading in Rust instead of the value the UPDATE stored.
When the two reads straddled a second boundary, every lease-gated
finalizer (JOB-2) for that job silently no-oped and the row hung at
'running' until recover_stuck_jobs. The UPDATE now uses RETURNING and
the struct carries the stored value.

Panic containment. A panicking handler future unwound through
process_next_job, killing the whole worker loop (Supervised restarts it
with exponential backoff, so processing stalled repeatedly), and since
no finalizer ran, the row hung at 'running'. A deterministically
panicking payload looped forever: the attempts >= max_attempts check
lives in exactly the code the panic skipped. The handler future is now
wrapped in catch_unwind (still directly owned by tokio::time::timeout,
preserving cancel-on-timeout) and a panic feeds the normal retry/fail
branching with the panic reason recorded.

Tenant-shard job processing. In sharded mode, ingestion enqueues jobs
into the tenant shard the request resolved (raw.rs / extract.rs), but
the single worker only polled the monolith: shard jobs sat pending
forever. The worker now also drains RESIDENT active shards every tick
(snapshot_all_handles: no loads, no touches, so polling cannot pin
tenants resident) and sequentially sweeps ALL active tenants on the
5-minute stuck-recovery cadence to catch shards evicted with a backlog.
The ingestion.fact_extract / ingestion.entity_extract handlers now
resolve the shard from the payload's user_id at execution time instead
of closing over the monolith: per-file autoincrement memory ids collide
across shards, so the old fixed-db lookup could derive facts against
another tenant's memory row.

Tests: struct lease token equals stored claimed_at; panicking handler
retries then fails terminally with the reason recorded; a job enqueued
into a tenant shard is drained end-to-end by the worker.

comment-check-skip: any flagged declarations are pre-existing; all
declarations added by this change are documented.
@Ghost-Frame
Ghost-Frame merged commit 22b65a7 into main Jul 18, 2026
12 checks passed
@Ghost-Frame
Ghost-Frame deleted the fix/jobs-correctness branch July 18, 2026 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant