fix(jobs): true lease token, panic containment, tenant-shard job processing#184
Merged
Conversation
…essing Three fixes to the durable jobs pipeline: Lease token integrity. claim_next_job populated Job.claimed_at from a second clock reading in Rust instead of the value the UPDATE stored. When the two reads straddled a second boundary, every lease-gated finalizer (JOB-2) for that job silently no-oped and the row hung at 'running' until recover_stuck_jobs. The UPDATE now uses RETURNING and the struct carries the stored value. Panic containment. A panicking handler future unwound through process_next_job, killing the whole worker loop (Supervised restarts it with exponential backoff, so processing stalled repeatedly), and since no finalizer ran, the row hung at 'running'. A deterministically panicking payload looped forever: the attempts >= max_attempts check lives in exactly the code the panic skipped. The handler future is now wrapped in catch_unwind (still directly owned by tokio::time::timeout, preserving cancel-on-timeout) and a panic feeds the normal retry/fail branching with the panic reason recorded. Tenant-shard job processing. In sharded mode, ingestion enqueues jobs into the tenant shard the request resolved (raw.rs / extract.rs), but the single worker only polled the monolith: shard jobs sat pending forever. The worker now also drains RESIDENT active shards every tick (snapshot_all_handles: no loads, no touches, so polling cannot pin tenants resident) and sequentially sweeps ALL active tenants on the 5-minute stuck-recovery cadence to catch shards evicted with a backlog. The ingestion.fact_extract / ingestion.entity_extract handlers now resolve the shard from the payload's user_id at execution time instead of closing over the monolith: per-file autoincrement memory ids collide across shards, so the old fixed-db lookup could derive facts against another tenant's memory row. Tests: struct lease token equals stored claimed_at; panicking handler retries then fails terminally with the reason recorded; a job enqueued into a tenant shard is drained end-to-end by the worker. comment-check-skip: any flagged declarations are pre-existing; all declarations added by this change are documented.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lease token integrity.
claim_next_jobpopulatedJob.claimed_atfrom a second clock reading in Rust instead of the value the UPDATE stored. When the two reads straddled a second boundary, every lease-gated finalizer (JOB-2) silently no-oped and the row hung at 'running' untilrecover_stuck_jobs. The UPDATE now usesRETURNINGand the struct carries the stored value.Panic containment. A panicking handler future unwound through
process_next_job, killing the whole worker loop, and since no finalizer ran, the row stayed 'running'. A deterministically panicking payload looped forever: theattempts >= max_attemptscheck lives in exactly the code the panic skipped. The handler future is now wrapped incatch_unwind— still directly owned bytokio::time::timeout, preserving cancel-on-timeout — and a panic feeds the normal retry/fail branching with the reason recorded.Tenant-shard job processing. In sharded mode, ingestion enqueues jobs into the tenant shard the request resolved, but the worker only polled the monolith: shard jobs sat pending forever. The worker now drains resident active shards every tick (
snapshot_all_handles— no loads, no touches, so polling cannot pin tenants resident or fight eviction) and sequentially sweeps all active tenants on the 5-minute stuck-recovery cadence to catch shards evicted with a backlog. Theingestion.fact_extract/ingestion.entity_extracthandlers resolve their database from the payload'suser_idat execution time instead of closing over the monolith — per-file autoincrement memory ids collide across shards, so the fixed-db lookup could derive facts against another tenant's memory row.Testing
claimed_at_in_struct_matches_stored_rowpanicking_handler_is_retried_not_stuck/panicking_handler_fails_after_max_attempts(panic reason recorded on the row)worker_drains_resident_tenant_shard_jobs(end-to-end: realTenantRegistry, job enqueued into a shard, worker drains it)cargo test -p kleos-lib -p kleos-server --lib jobs: 19 + 1 passed.