feat: optimize Bull background job processing

[JemimahEkong]

Summary

This PR improves background job processing throughput and reliability by introducing centralized queue management, priority-based scheduling, dead-letter handling, retry strategy abstraction, queue metrics monitoring, and load-testing infrastructure.

The implementation focuses on reducing queue latency, improving failure recovery, and providing better operational visibility into BullMQ-based processing.

Closes #525

Changes

Queue Architecture

• Added centralized QueueModule for Bull queue registration and configuration
• Added QueueService for standardized job submission and queue interaction
• Centralized queue initialization previously spread across modules

Priority Queue Processing

• Added Bull-native priority mapping through PrioritizationService
• Implemented priority-aware job scheduling
• Added support for configurable job priority levels

Dead Letter Queue

• Added dedicated dead-letter queue infrastructure
• Added DeadLetterService for failed-job capture
• Preserved failure metadata for investigation and recovery

Retry Strategy Optimization

• Added injectable RetryStrategyService
• Centralized retry configuration management
• Enabled consistent retry behavior across queue consumers

Worker Integration

• Added worker bridge layer to connect existing workers to Bull processors
• Added configurable concurrency settings for queue consumers
• Added graceful queue shutdown handling

Monitoring & Metrics

• Added queue metrics collection service
• Added queue depth monitoring
• Added processing-time metrics
• Added periodic queue health polling

Testing & Validation

• Added load-testing benchmark infrastructure
• Added throughput benchmark script for queue performance validation
• Verified queue registration, worker wiring, retry behavior, and dead-letter handling

Documentation

• Expanded queue documentation with architecture and configuration guidance
• Documented queue processing flow, retry handling, priorities, and monitoring

Files Added

• src/queues/queue.module.ts
• src/queues/queue.service.ts
• src/queues/dead-letter/*
• src/queues/retry/*
• src/queues/metrics/*
• src/workers/bridge/*
• tests/load/queue-throughput.benchmark.ts

Files Updated

• src/app.module.ts
• src/common/constants/queue.constants.ts
• src/messaging/messaging.module.ts
• src/monitoring/metrics/metrics-collection.service.ts
• src/queues/prioritization/prioritization.service.ts
• src/queues/README.md

Acceptance Criteria

• Priority queue implemented
• Dead letter queue for failed jobs
• Job retry strategy optimized
• Memory and queue metrics monitored
• Load-test infrastructure added
• Queue configuration documented

Notes

• Throughput benchmarking requires a running Redis instance and can be executed using the included benchmark script.
• Changes are additive and focused on queue infrastructure, with no application-facing behavior changes.

[drips-wave]

@JemimahEkong Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits