Shared Subscription Throughput Bottleneck: 400K msg/s Published, Only 20-40K msg/s Consumed

[wxhdavid]

Summary

In a large-scale IoT benchmark with 10 million concurrent MQTT connections, all devices publish status messages matching a single $share group subscription. The publish side sustains 400K+ msg/s, but the subscribe side only consumes 20-40K msg/s (5-10%). The dist-service layer is the confirmed bottleneck.

Environment

BifroMQ Cluster (15 nodes):

• Broker: 7 nodes, 48 vCPU each — 1-5% CPU, idle
• Dist: 5 nodes, 48 vCPU each — 64-79% CPU, near saturation
• Retain: 3 nodes, 8 vCPU each — 0-1% CPU, idle

Load Generator: 25 machines x 400K connections = 10M total, 8 NICs per machine, custom Rust MQTT client.

Subscriber: 13 machines x 30 connections = 390 subscribers, all in one $share group.

BifroMQ Version: 4.0.0-incubating, JDK 25, Linux (Alibaba Cloud ECS)

Topic Structure:

Publish:   /device_status/{device_id}/{product_key}/{product_model}/{region}/
Subscribe: $share/group_name//device_status/#

Test Results

Config A — 430,000 publishers, 1 msg/s each

• Pub QPS: 431K (peak at 8.4M connections)
• Sub QPS: 4-6K/s

Config B — 5,000 publishers, 86 msg/s each

• Pub QPS: 424K (zero drops)
• Sub QPS: 4K/s

Config C — 100,000 publishers, 4 msg/s each

• Pub QPS: 389K (zero drops at low connection count)
• Sub QPS: 22-42K/s (best result)

Config D — Same as C, observed at only 290K connections

• Pub QPS: 389K, zero drops, zero ConnFail, broker 1-5% CPU
• Sub QPS: 22K/s — proves the ceiling exists regardless of broker load

Key Observations

1. Broker nodes nearly idle (1-5% CPU) — MQTT protocol handling is not the bottleneck.

2. Dist nodes are the bottleneck at 64-79% CPU. All 5 dist nodes show load avg 30-38 on 48-core machines.

3. Sub throughput scales with publisher diversity — 5K unique publishers gives 4K/s sub; 100K unique publishers gives 22-42K/s sub. This suggests dist-layer partitioning doesn't distribute load evenly when message sources are concentrated.

4. Subscriber QoS 0 vs QoS 1 makes no difference — bottleneck is in the dist-layer fanout/delivery pipeline, not in broker-to-subscriber PUBACK.

5. Payload size reduction (900B to 100B) did not improve sub throughput.

6. At low connection count (290K), with zero drops, sub is still capped at 22K/s — the shared subscription delivery ceiling is independent of broker load.

Questions

1. Is there a recommended way to tune shared subscription throughput for high fan-in scenarios (many publishers, single $share group, few hundred subscribers)?

2. Would increasing dist_worker_fanout_parallelism help? Default is max(2, cpus/8) = 6 on 48-core machines — seems conservative for 400K+ msg/s.

3. Are there other system properties we should tune? (e.g. data_plane_burst_latency_ms, dist_worker_inline_fanout_threshold)

4. Is the single-threaded DeliverExecutor design intentional for ordering? Could it be relaxed for unordered shared subscriptions?

Detailed source code analysis and full optimization history in comments below.

[wxhdavid]

Source Code Analysis — Delivery Pipeline Bottlenecks

After reading the BifroMQ source code, we traced the full delivery path for shared subscriptions:

DistWorkerCoProc.batchDist()
  -> routeCache.get(tenantId, topic)          // Caffeine async cache
    -> returns Set<Matching> including GroupMatching
  -> DeliverExecutorGroup.submit(tenantId, routes, topicMsgPack)
    -> send(GroupMatching, ...)
      -> ThreadLocalRandom picks 1 subscriber   // unordered shared sub
      -> DeliverExecutor.submit(route, msgPack, inline)
        -> single-threaded ThreadPoolExecutor    // bottleneck 1
          -> BatchCallScheduler.schedule()
            -> Batcher.submit()
              -> IDeliverer.deliver() RPC        // bottleneck 2

Bottleneck 1: Too few fanout executors, each single-threaded

DistFanOutParallelism (DistFanOutParallelism.java:33):

super("dist_worker_fanout_parallelism",
    Math.max(2, EnvProvider.INSTANCE.availableProcessors() / 8), IntegerParser.POSITIVE);

On 48-core dist nodes, this defaults to 6 executors.

DeliverExecutor (DeliverExecutor.java:54-57):

executor = new ThreadPoolExecutor(1, 1, 0L, TimeUnit.MILLISECONDS,
    new LinkedTransferQueue<>(),
    EnvProvider.INSTANCE.newThreadFactory("deliver-executor-" + id));

Each executor is single-threaded. So the entire 48-core dist node has only 6 threads doing actual delivery work.

Bottleneck 2: Delivery batching with back-pressure

MessageDeliverer (MessageDeliverer.java:37-38):

super(batchDeliveryCallBuilderFactory,
    Duration.ofMillis(DataPlaneMaxBurstLatencyMillis.INSTANCE.get()).toNanos()); // default 5000ms

The BatchCallScheduler batches delivery calls and has back-pressure. Under high load, calls are rejected with BACK_PRESSURE_REJECTED.

Bottleneck 3: Hash distribution across executors

DeliverExecutorGroup.send() (DeliverExecutorGroup.java:280-285):

int idx = route.hashCode() % fanoutExecutors.length;

390 subscribers hashed into 6 executors. Uneven hash distribution can create hotspots.

Throughput estimation

With 6 single-threaded executors, assuming 1ms per delivery RPC:

• Unbatched: 6 x 1000 = 6,000 msg/s
• With batching (50 msgs/batch): 6 x 1000 x 50 = 300,000 msg/s (ideal)

Our observed 22-42K/s suggests effective batch size or RPC latency limits throughput well below the ideal.

[wxhdavid]

Full Optimization History

What we tried on the client/load-generator side

• Increased publisher max_inflight from 2 to 50, channel_capacity from 5 to 100 — eliminated publisher drops
• Changed subscriber QoS from 1 to 0 — no improvement on sub throughput
• Increased subscriber channel_capacity to 1000
• Varied publisher count: 5K / 50K / 100K unique publishing devices — 100K gave best sub throughput
• Reduced payload from ~900 bytes to ~100 bytes — no sub-side improvement
• Lowered connect_rate from 40,000/s to 5,000-10,000/s to prevent broker overload during ramp-up
• Prioritized publisher connections (publishers connect first) to reach target QPS quickly

JVM parameters we plan to try on dist nodes

-Ddist_worker_fanout_parallelism=24
-Ddata_plane_burst_latency_ms=10000

We haven't tried these yet and would appreciate guidance on whether these are safe to increase and what side effects to expect.

BifroMQ plugin settings

• ENABLE_FAST_AUTH=true (bypass remote auth check)
• SKIP_EVENTS_FOR_TEST=true (drop all events during benchmark)

Version

• BifroMQ: 4.0.0-incubating
• JDK: 25
• OS: Linux (Alibaba Cloud ECS)
• Load generator: Custom Rust client (rumqttc 0.24)

Any guidance on tuning the dist-layer delivery pipeline for this workload would be greatly appreciated. Thank you!