[Suggestion] Use cluster in EC2 or Autoscalegroup

[ko2024]

RabbitMQ series

4.2.x

Operating system (distribution) used

Ubuntu

How is RabbitMQ deployed?

Debian package

What would you like to suggest for a future version of RabbitMQ?

With the change of metadata store to Khepri, I observed changes in cluster formation due to the use of Raft, and with ASG, if the machine quorum is lost, the cluster becomes unavailable. For version 4.2, do you still recommend the use of AutoScaleGroup?

[michaelklishin]

@ko2024 Khepri is orthogonal to peer discovery.

With any Raft-based system in RabbitMQ, losing a majority of online nodes means losing cluster availability for writes. This is not any different in 4.2 vs 4.3 (main).

AWS allows autoscaling groups to be configured with a MinSize, a DesiredCapacity and the maximum number of node replacements in flight (the MaxBatchSize setting). And then there is InstanceMaintenancePolicy.

The end goal is to not lose an online majority, so you might choose to go with 5 or 7 nodes in environments where the loss of 2 nodes is likely.

@the-mikedavis @lukebakken are on the RabbitMQ engineering team at AWS, so I'll let them correct me.

[michaelklishin]

I guess the question here might be about the general use of the peer discovery "related" destructive features.

The docs should clearly highlight the risks.

Why do we have those features in the peer discovery subsystem? Because they were in place in the original community plugin that was very ASG-centric (in my opinion) from the start, and we've folded it into RabbitMQ core largely as it was.

But we did document the risks.

[the-mikedavis]

There aren't many details in the post about how you are using ASG and RabbitMQ. Could you expand on your usage pattern and why you are wondering if ASG is a good idea or not?

if the machine quorum is lost, the cluster becomes unavailable

This is by design (i.e. Raft) and ensures consistency. Is there a specific ASG behavior that triggers loss of quorum that you are trying avoid?

Very generally I can say that ASG is a pretty flexible tool for EC2 management and you'd be hard-pressed to do better with the EC2 APIs alone. Though I know of a few rough-edges around cross-AZ instance replacements that can make state-management hard.

[ko2024]

Thank you very much for the information; it clarified my doubt.

I'm not having any problems using ASG, but my doubt was about whether there was any guidance due to the change to Raft in version 4.2 and the upcoming version 4.3.

I really like using ASG because of the benefits it offers compared to EC2.