Expose a Store Node Eviction API via APIServer

[syh194194]

Is your feature request related to a problem? Please describe.

When a BifroMQ cluster node fails due to disk exhaustion or partial storage corruption, the SWIM failure detector can be stalled indefinitely. If the OS is still responsive enough to ACK TCP probes while the application layer is hung, the failure detector repeatedly cancels its suspicion timer and never
declares the node dead. As a result, the Raft voter list is never cleaned up, write operations stall, and the cluster cannot self-heal — regardless of whether the node is removed from the load balancer. The only reliable fix today is to physically power off the machine and wait up to ~90 seconds for the
automatic pipeline to complete. There is no operator-facing API to accelerate or override this.

Re-deploying on the same IP makes things significantly worse: the new process gets a new UUID-based identity while the old identity persists in the CRDT membership, causing other nodes' indirect probes to keep the old identity alive. The cluster ends up stuck with a ghost voter that can never be evicted.

Describe the solution you'd like

Two new endpoints in bifromq-apiserver (default port 8091):

1. Evict a store node from all its Raft groups

DELETE /store/node
Headers:
store_name: dist.worker | inbox.store | retain.store
store_id:

This triggers a ChangeConfigCommand against every KV range for which the target store is a voter or learner, removing it immediately. This is exactly what UnreachableReplicaRemovalBalancer does automatically, but driven by operator intent rather than a timeout.

Relevant internal code: UnreachableReplicaRemovalBalancer.java, KVStoreBalanceController.java

2. Evict a node from base-cluster membership

DELETE /cluster/node
Headers:
host:
port:

This bypasses the suspicion timeout and directly removes the target HostEndpoint from CRDT membership, unblocking the isMissingInStore() gate that UnreachableReplicaRemovalBalancer depends on.

Relevant internal code: AutoDropper.java, HostMemberList.java

Describe alternatives you've considered

• Tuning timeouts (zombieProbeDelayInMS, suspicionMultiplier, baseProbeInterval): Reduces the self-healing window in normal failure cases, but does not help when the node is half-dead and the failure detector keeps getting suppressed by sporadic ACKs.
• Custom balancer plugin: A custom balancer can produce ChangeConfigCommand, but it still cannot bypass the isMissingInStore() gate without also evicting the node from base-cluster membership. It also requires custom code deployment rather than a simple API call.
• Graceful shutdown on the failed node: Only works if the node is reachable enough to run AgentHost.close(), which is not the case when the disk is full or the process is unresponsive.

Additional context

The internal command types (ChangeConfigCommand, QuitCommand, RecoveryCommand) and the balancer framework already exist in base-kv-store-balance-spi. The apiserver handler pattern is well-established in bifromq-apiserver/src/main/java/org/apache/bifromq/apiserver/http/handler/. The main work is wiring an
external trigger through to these existing internal mechanisms.

[syh194194]

Root Cause Chain (Failure Cascade)

• Trigger: Disk Full (RocksDB State Corruption)

• Abnormal State: Process becomes "Half-Dead" (Sporadic TCP ACKs)

• Detection Failure: SWIM failure detector is repeatedly suppressed -> Never declares the node as "Fail/Dead".

• Self-Healing Blocked: Cluster landscape remains unchanged -> UnreachableReplicaRemovalBalancer is not triggered.

• Consensus Pollution: Raft voter list persistently retains the corrupted storeId.

• Write Path Stalled: Write operations are blocked (Majority commit failure / Inflight window full / Append timeout).

• 1. Ineffective Mitigation A: Removing the node from the Load Balancer is useless (LB is not part of the CRDT cluster membership).

• 2. Ineffective Mitigation B: Re-deploying on the same IP exacerbates the issue (Old identity persists + New identity is not in Raft + IP reuse causes false-positive probe ACKs).

• Current Sole Workaround: Hard Shutdown -> SWIM definitively detects failure -> Balancer self-healing pipeline is fully unblocked.

[Gujiawei-Edinburgh]

Root Cause Chain (Failure Cascade)

• • Trigger: Disk Full (RocksDB State Corruption)* Abnormal State: Process becomes "Half-Dead" (Sporadic TCP ACKs)* Detection Failure: SWIM failure detector is repeatedly suppressed -> Never declares the node as "Fail/Dead".* Self-Healing Blocked: Cluster landscape remains unchanged -> UnreachableReplicaRemovalBalancer is not triggered.* Consensus Pollution: Raft voter list persistently retains the corrupted storeId.* Write Path Stalled: Write operations are blocked (Majority commit failure / Inflight window full / Append timeout).* [ ]
    [ ] 1. Ineffective Mitigation A: Removing the node from the Load Balancer is useless (LB is not part of the CRDT cluster membership).* [ ]
    [ ] 2. Ineffective Mitigation B: Re-deploying on the same IP exacerbates the issue (Old identity persists + New identity is not in Raft + IP reuse causes false-positive probe ACKs).* Current Sole Workaround: Hard Shutdown -> SWIM definitively detects failure -> Balancer self-healing pipeline is fully unblocked.

Thanks for reaching out. I think exposing internal mechanism as OpenAPI is risky.

From an operational perspective, disk exhaustion is also considered a preventable infrastructure condition. In practice, we expect deployments to rely on monitoring, alerting, and capacity management to avoid entering this corrupted/half-dead state in the first place.

[popduke]

Agree with @Gujiawei-Edinburgh, I don't think adding a manual eviction API is the right direction here.

SWIM should not be treated as a reliable detector for every kind of half-dead process. It mainly works around reachability: if a node stops responding to probes, it can eventually be suspected and marked dead. If the process is still alive enough to answer probes but the application is stuck because of disk full, storage corruption, or resource exhaustion, that should be caught by normal node and service monitoring much earlier.

I also don't agree that redeploying on the same IP necessarily leaves a ghost voter that can never be evicted. The latest code already has logic to detect and clear stale members with the same address and port but a different identity. If this still fails, we need a reproducible simulation test showing the exact stuck ghost-voter case.

About the "~90 seconds" recovery time: for a disk-full, corrupted, half-dead, or physically powered-off node, automatic recovery within around 90 seconds is not an unreasonable MTTR. A forceful API feels like the wrong tradeoff unless there is a reproducible case showing that the latest code cannot self-heal after the failed process is actually stopped.

[syh194194]

Thanks for the thoughtful feedback. Several of your points are well-taken, and I want to refine the request based on them.

On the reproducibility question

You're right that this FR is grounded in static code analysis rather than a reproducible end-to-end test. Specifically, the "ghost voter that can never be evicted after same-IP redeploy" claim deserves verification: I see that AutoDropper.startSuspicion() already converts Fail to Quit when isZombie()
detects same-address-different-id, which should handle the redeploy case once the failure detector fires. I'll work on a reproducible simulation (kill -9 mid-write, redeploy with cleared data, observe whether the cluster self-heals) before pushing this further. If the latest code does recover within
~90s, the redeploy concern is moot.

On the "half-dead" stall

I agree that detecting application-layer hangs (disk full, RocksDB stuck) is fundamentally outside SWIM's contract — SWIM probes reachability, not liveness. Where I'd push back gently: when the OS is responsive enough to ACK probes but the app is hung, today's BifroMQ has no signal at all that something
is wrong. Tuning zombieProbeDelayInMS doesn't help, and the operator's only tool is power off. If this category is truly out of scope for SWIM, maybe what's missing isn't an eviction API but an app-layer probe contract (e.g., a self-test executed by the local node that, on failure, calls
IAgentHost.shutdown() so SWIM can detect it normally). Would such a proposal be more aligned with the project's direction?

On the 90s MTTR bound

Fair. I'll narrow the FR's scope: rather than a forceful eviction API, the immediate operational gap is observability. There is currently no easy way for an operator to answer "is the cluster self-healing right now, and how long until it's done?" — GET /cluster and GET /store/landscape show the current
state but not the in-flight suspicion timers, balancer decisions, or expected ETA. I'll close this FR and open a smaller observability-focused one.

One documentation clarification

The "removing from LB has no effect" point in the original report wasn't an argument for the eviction API — it was a common operator misconception that BifroMQ's docs don't currently address. Could the project consider adding a short note clarifying that base-cluster membership is decoupled from the
external load balancer? That alone would have prevented hours of confusion in our incident.

[popduke]

On observability: BifroMQ already exposes enough metrics with the basecluster.* prefix. Maybe this was missed in your static code analysis, but I think these metrics should be enough for the visibility you mentioned.

On the LB clarification: I don't remember BifroMQ docs saying that the BifroMQ cluster has anything to do with the external LB. Could you point me to the wording in the docs or elsewhere that may mislead users? Then we can review and fix that part if needed.