fix(queue): enforce idempotent consumption with unique keys#1
Open
tobechi-dev wants to merge 3 commits into
Open
fix(queue): enforce idempotent consumption with unique keys#1tobechi-dev wants to merge 3 commits into
tobechi-dev wants to merge 3 commits into
Conversation
- Add IdempotentConsumer class for at-least-once message delivery - Use idempotency_keys table with unique constraints and upserts - Check cache before processing, store result after completion - Skip reprocessing for duplicate messageIds within TTL window - Add unit tests for core idempotent processing flows - Add integration tests for duplicate message handling Branch: fix/queue-idempotency-fresh
- Document problem: at-least-once delivery causing duplicates - Solution architecture: write-layer deduplication with unique keys - Implementation details: flow diagrams, class methods, configuration - Why chosen approach over alternatives (locks, client-side) - Files created (service, tests) and edited (bug fix) - Usage examples for queues, webhooks, scheduled jobs - Performance considerations and future enhancements
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements idempotent consumer to prevent duplicate side effects when messages are redelivered or processed concurrently under at-least-once delivery.
Changes
New Files
src/services/idempotentConsumer.ts- Core IdempotentConsumer classsrc/__tests__/idempotentConsumer.test.ts- Unit teststests/integration/idempotentConsumer.test.ts- Integration testsdocs/idempotent-consumer.md- Technical documentationBug Fix
src/db/repositories/idempotencyRepository.ts- Fixed JSON parse for responseBodyImplementation Details
The idempotent consumer uses write-layer deduplication:
idempotency_keystable before processingThis ensures messages are processed exactly once even when:
Testing
All tests pass:
Related Issue
Fixes CredenceOrg#231 - [Fresh 2026-04][Backend] Queues: idempotent consumer under at-least-once delivery (dedupe at write layer)