Skip to content

Prevent stale workers from acknowledging newer message claims #22

Description

@polyrand

Context

LiteQueue can retry a locked message, but a claim has no identity beyond message_id. done(), mark_failed(), and retry() update any matching row without checking its current state.

Current code:

  • pop() changes READY to LOCKED and returns the message.
  • retry() changes any matching message to READY and leaves its old lock_time in place.
  • done() and mark_failed() accept acknowledgements from any worker that knows the message ID.

A focused reproduction on commit 2b82ede showed this race:

  1. Worker A pops a message.
  2. Recovery retries the message.
  3. Worker B pops the same message.
  4. Worker A calls done(message_id).
  5. LiteQueue marks worker B's active claim as done.

The stale done() returned True, and the newer claim ended in DONE.

Why implement this

Retry is meant to recover work after a worker stops responding. Without claim ownership, a slow worker can acknowledge a newer delivery. If the newer worker then fails, LiteQueue has already removed the task from active work. This breaks reliable redelivery and can lose work.

Proposed direction

  • Add a claim token or monotonically increasing attempt number to each successful pop.
  • Include the claim identity in the returned Message or a dedicated claim type.
  • Require message_id, claim identity, and LOCKED status in done() and mark_failed() updates.
  • Restrict retry to documented source states.
  • Clear obsolete lock_time and done_time fields when returning work to READY.
  • Provide one atomic operation for retrying expired claims instead of requiring list_locked() followed by separate retry() calls.
  • Document and migrate the schema and acknowledgement API.

Acceptance criteria

  • A stale worker cannot complete or fail a newer claim.
  • Invalid transitions return a clear failure without changing the row.
  • Retrying a claim clears obsolete timestamps.
  • Expired-claim recovery is atomic under concurrent workers.
  • Tests reproduce the stale-worker race for both pop implementations.
  • make test passes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions