Fix duplicate plan limit overage emails by niemyjski · Pull Request #2269 · exceptionless/Exceptionless

niemyjski · 2026-05-29T23:54:58Z

Root cause

Every web pod registers EnqueueOrganizationNotificationOnPlanOverage at startup. Foundatio pub/sub delivers each PlanOverage message to all subscribers, so a single monthly overage event enqueued one work item per running web pod (e.g. 6 pods → 6 identical items).

The previous ThrottlingLockProvider(slotsPerPeriod: 1, period: 1 hour) allowed exactly one item through per calendar-hour bucket. When a duplicate item lost the lock race it was abandoned back to the queue (not discarded). Once the next hour bucket opened the item was reprocessed and acquired a fresh lock — producing one email per hour for each duplicate, matching the reported six-emails-over-a-day pattern.

The TimeSpan.FromMinutes(15) in the old GetWorkItemLockAsync was the work-item processing timeout (how long the lock was held during execution), not the throttle window — these are independent parameters.

Could the bot-cleanup job have retriggered the edge? Unlikely: bot event deletion removes documents from Elasticsearch but does not decrement the Redis usage counters that IncrementTotalAsync uses for edge detection, so the monthly overage edge would not re-fire from cleanup.

Fix

Two independent layers, both required:

1. Queue-level dedup — OrganizationNotificationWorkItem implements IHaveUniqueIdentifier and DuplicateDetectionQueueBehavior<WorkItemData> is registered in Bootstrapper. The unique identifier is Organization:{orgId}:notification:{type} (via GetNotificationKey). Fanout enqueues from all pods collapse to a single queue entry.

2. Handler-level idempotency — In OrganizationNotificationWorkItemHandler:

Per-org distributed lock (30 min lease) serialises concurrent processing. The 30-minute lease comfortably exceeds the maximum realistic send-loop duration, eliminating the race window where a second worker could acquire a lock before the sent marker is written.
24-hour sent marker (Organization:{orgId}:notification:monthly-sent) ensures stale duplicates already in the queue at deploy time cannot retrigger an email.
Hourly items short-circuit at GetWorkItemLockAsync (return null lock) so they never occupy the lock/sent-key path and cannot suppress a later monthly notification.

Known limitations (acceptable trade-offs, documented in code comments):

If SendOverageNotificationsAsync throws mid-loop, some recipients will already have received the email and will receive it again on retry. This is intentional: suppressing retries on partial failure would silently skip un-notified users.
The 24h sent window is not billing-period aware. An overage within 24h of a billing-cycle reset could be delayed. Follow-up if this becomes an issue.

Changes

File	What changed
`OrganizationNotificationWorkItemHandler.cs`	Replaced `ThrottlingLockProvider` with handler lock + 24h sent marker; hourly items bypass email path; 30-min lock lease; class XML doc with RCA
`OrganizationNotificationWorkItem.cs`	Added `IHaveUniqueIdentifier`, `NotificationType` constants, `GetNotificationKey` static helper; removed all legacy key helpers
`Bootstrapper.cs`	Registered `DuplicateDetectionQueueBehavior<WorkItemData>` via DI
`OrganizationNotificationWorkItemHandlerTests.cs`	RCA-pinning unit tests: fanout without/with dedup, UniqueIdentifier format, legacy throttle regression
`OrganizationNotificationWorkItemHandlerIntegrationTests.cs`	Integration tests against real handler: dedup across hours, per-org isolation, 24h resend, hourly-only no-send, hourly-before-monthly, sent-marker idempotency
`CountingMailer.cs`	Thread-safe test mailer for counting/asserting sends

Testing

dotnet test tests/Exceptionless.Tests/ -- --filter-class Exceptionless.Tests.Jobs.WorkItemHandlers.OrganizationNotificationWorkItemHandlerTests
dotnet test tests/Exceptionless.Tests/ -- --filter-class Exceptionless.Tests.Jobs.WorkItemHandlers.OrganizationNotificationWorkItemHandlerIntegrationTests
dotnet test tests/Exceptionless.Tests/ -- --filter-class Exceptionless.Tests.Services.UsageServiceTests

All 10 notification tests pass. Full build is clean.

Breaking changes

None.

Copilot

Pull request overview

This PR addresses duplicate “plan limit overage” organization emails by adding queue-level duplicate detection (via a stable work-item unique identifier) and by tightening handler-side suppression logic so stale duplicates can’t trigger extra monthly overage emails later.

Changes:

Add a stable UniqueIdentifier to OrganizationNotificationWorkItem to enable cross-pod/work-queue deduplication.
Update OrganizationNotificationWorkItemHandler to (a) ignore hourly-only items and (b) suppress repeat monthly sends using a per-organization 24h “monthly-sent” cache marker plus a monthly-only lock.
Add regression tests covering delayed duplicate processing, hourly-then-monthly ordering, org isolation, and queue dedup behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/Exceptionless.Tests/Mail/CountingMailer.cs	Adds a test mailer that records organization-notice sends for assertions.
tests/Exceptionless.Tests/Jobs/WorkItemHandlers/OrganizationNotificationWorkItemHandlerTests.cs	Adds regression tests for duplicate enqueue/processing and correct monthly notification behavior.
src/Exceptionless.Core/Models/WorkItems/OrganizationNotificationWorkItem.cs	Implements `IHaveUniqueIdentifier` to provide a stable dedup key per org + overage type.
src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationNotificationWorkItemHandler.cs	Reworks handler throttling/suppression: monthly-only lock + 24h “sent” marker; hourly-only items no longer suppress monthly.
src/Exceptionless.Core/Bootstrapper.cs	Registers `DuplicateDetectionQueueBehavior<WorkItemData>` and wires queue behaviors into queue creation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

niemyjski · 2026-05-30T01:15:22Z

    public required bool IsOverHourlyLimit { get; init; }
    public required bool IsOverMonthlyLimit { get; init; }
+
+    public string? UniqueIdentifier => $"org-notification:{OrganizationId}:{(IsOverMonthlyLimit ? "monthly" : "hourly")}";


This should be Organization:{OrganizationId}:notification:{(IsOverMonthlyLimit ? "monthly" : "hourly")}"; ?

I normalized the notification type helpers, but I kept UniqueIdentifier on the deployed org-notification:{org}:{type} key so mixed-version web nodes still share the same queue dedupe marker during rollout.

Correct — the UniqueIdentifier is now Organization:{orgId}:notification:{type} (monthly or hourly) via GetNotificationKey. All legacy key helpers removed.

niemyjski · 2026-05-30T02:19:03Z

Updated this PR with the deeper RCA and coverage. The most likely failure mode is: (1) every web pod subscribes to PlanOverage, (2) a monthly PlanOverage fans out to each pod and each pod enqueues the same notification work item, and (3) the old ThrottlingLockProvider(1/hour) buckets by hour, so once duplicate monthly work items existed it could leak one email per hour bucket. I couldnt support the bot-cleanup theory from the code because removing bot events doesnt decrement organization usage totals, and monthly PlanOverage is edge-triggered in UsageService.IncrementTotalAsync. The updated tests now cover the fanout path, the legacy hourly leak, the real handler suppression path, hourly-before-monthly ordering, 24-hour resend behavior, and legacy sent-key compatibility.

niemyjski · 2026-05-30T02:56:02Z

+        services.ReplaceSingleton<ICacheClient>(sp => new InMemoryCacheClient(new InMemoryCacheClientOptions
+        {
+            TimeProvider = sp.GetRequiredService<TimeProvider>(),
+            LoggerFactory = sp.GetRequiredService<ILoggerFactory>()
+        }));
+
+        services.ReplaceSingleton<IMessageBus>(sp => new InMemoryMessageBus(new InMemoryMessageBusOptions
+        {
+            Serializer = sp.GetRequiredService<ISerializer>(),
+            TimeProvider = sp.GetRequiredService<TimeProvider>(),
+            LoggerFactory = sp.GetRequiredService<ILoggerFactory>()
+        }));
+
+        services.ReplaceSingleton<IMessagePublisher>(sp => sp.GetRequiredService<IMessageBus>());
+        services.ReplaceSingleton<IMessageSubscriber>(sp => sp.GetRequiredService<IMessageBus>());


this should already be the default, why are we registering it again?

niemyjski · 2026-05-30T02:56:13Z

+    }
+
+    [Fact]
+    public async Task RunAsync_WhenOnePlanOverageIsObservedBySixSubscribersWithQueueDedup_ShouldEnqueueOneWorkItem()


three part name.. check pr

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

+    public override Task<ILock?> GetWorkItemLockAsync(object workItem, CancellationToken cancellationToken = default)
+    {
+        var wi = (OrganizationNotificationWorkItem)workItem;
+        if (!ShouldSendNotificationEmail(wi))
+            return Task.FromResult<ILock?>(null);
+
+        return _lockProvider.TryAcquireAsync(GetLegacyNotificationLockKey(wi.OrganizationId, wi.NotificationType), TimeSpan.FromMinutes(15), cancellationToken);
+    }


+    public static string GetLegacyNotificationLockKey(string organizationId, string notificationType)
+    {
+        return notificationType == OrganizationNotificationWorkItem.MonthlyNotificationType
+            ? $"{nameof(OrganizationNotificationWorkItemHandler)}:{organizationId}:{notificationType}-lock"
+            : GetNotificationLockKey(organizationId, notificationType);
+    }


+    public override Task<ILock?> GetWorkItemLockAsync(object workItem, CancellationToken cancellationToken = default)
+    {
+        var wi = (OrganizationNotificationWorkItem)workItem;
+        if (!ShouldSendNotificationEmail(wi))
+            return Task.FromResult<ILock?>(null);
+
+        return _lockProvider.TryAcquireAsync(GetLegacyNotificationLockKey(wi.OrganizationId, wi.NotificationType), TimeSpan.FromMinutes(15), cancellationToken);
+    }


+    public static string GetLegacyNotificationLockKey(string organizationId, string notificationType)
+    {
+        return notificationType == OrganizationNotificationWorkItem.MonthlyNotificationType
+            ? $"{nameof(OrganizationNotificationWorkItemHandler)}:{organizationId}:{notificationType}-lock"
+            : GetNotificationLockKey(organizationId, notificationType);
+    }


+    public override Task<ILock?> GetWorkItemLockAsync(object workItem, CancellationToken cancellationToken = default)
+    {
+        var wi = (OrganizationNotificationWorkItem)workItem;
+        if (!ShouldSendNotificationEmail(wi))
+            return Task.FromResult<ILock?>(null);
+
+        return _lockProvider.TryAcquireAsync(GetLegacyNotificationLockKey(wi.OrganizationId, wi.NotificationType), TimeSpan.FromMinutes(15), cancellationToken);
+    }


+    public static string GetLegacyNotificationLockKey(string organizationId, string notificationType)
+    {
+        return notificationType == OrganizationNotificationWorkItem.MonthlyNotificationType
+            ? $"{nameof(OrganizationNotificationWorkItemHandler)}:{organizationId}:{notificationType}-lock"
+            : GetNotificationLockKey(organizationId, notificationType);
+    }


Root cause: every web pod subscribes to PlanOverage at startup via EnqueueOrganizationNotificationOnPlanOverage. Foundatio pub/sub delivers each message to all subscribers, so a single monthly overage event enqueued one work item per running web pod. The original ThrottlingLockProvider(1/hour) allowed exactly one item through per calendar-hour bucket; abandoned duplicates were re-queued and reprocessed once each new bucket opened — producing one email per hour for each duplicate item. Fix: - Queue-level dedup: OrganizationNotificationWorkItem implements IHaveUniqueIdentifier and DuplicateDetectionQueueBehavior is registered so fanout enqueues collapse to one item. - Handler-level idempotency: per-org distributed lock (30 min) + 24-hour sent marker ensure stale duplicates already in the queue at deploy time cannot retrigger an email. - Hourly items short-circuit at GetWorkItemLockAsync and never enter the lock/sent-key path, preventing hourly overages from suppressing subsequent monthly notifications. Also add RCA-pinning unit tests (TestWithServices) and integration tests (IntegrationTestsBase) covering fanout dedup, legacy hourly throttle regression, per-org isolation, 24h resend window, hourly-before-monthly ordering, and idempotency via existing sent marker. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

+    {
+        // Arrange
+        using var workItemQueue = CreateWorkItemQueue(
+            new DuplicateDetectionQueueBehavior<WorkItemData>(CacheClient, GetService<ILoggerFactory>(), TimeSpan.FromHours(24)));


github-actions · 2026-05-30T03:25:02Z

Package	Line Rate	Branch Rate	Complexity	Health
Exceptionless.Insulation	25%	23%	203	❌
Exceptionless.Web	73%	62%	3922	➖
Exceptionless.AppHost	18%	9%	82	❌
Exceptionless.Core	69%	63%	7895	➖
Summary	68% (13651 / 19951)	62% (7191 / 11648)	12102	➖

github-code-quality Bot found potential problems May 29, 2026

View reviewed changes

niemyjski requested a review from Copilot May 30, 2026 01:03

niemyjski self-assigned this May 30, 2026

Copilot started reviewing on behalf of niemyjski May 30, 2026 01:03 View session