RabbitMQ Scalability Testing

RabbitMQ Scalability Testing

Scaling is the process of increasing or decreasing the capacity of the system by changing the number of processes available to service requests. So, what if your system is getting slow? How to validate if RabbitMQ cluster is still handling its load? Or maybe one of your services can’t consume events fast enough? In this post we will go through couple Spring Boot 2 + RabbitMQ system scalability testing scenarios.

Testing Environment

One Publisher – One Consumer

For this use case we will have two Spring Boot services started one as Publisher – it will send events to RabbitMQ and one as Consumer – listener on RabbitMQ exchange.

5000 messages

10000 messages

25000 messages

Results

For 5000 messages case the best performance was achieved on 2 nodes cluster with no sharding setup and the worst on both clusters with queues sharded into 4 shards per node. Results are really similar on all setups since amount of messages are quite small. 10000 messages case performance was better on 2 nodes setups vs on 6 nodes. 25000 messages – showed the best data processing difference per setup. 2 Nodes clusters was again performing better than 6 nodes. One producer and one consumer setup testing demonstrated that for this setup RabbitMq performs better with smaller amount of nodes. Sharding of the queue do not increase performance much because consumer can’t consume messages fast enough to make use of shards. Consumer application is the bottle neck.

One Publisher – Five Consumers

For this use case we will have six Spring Boot services started one as Publisher – it will send events to RabbitMQ and five as Consumers – listeners on RabbitMQ exchange.

5000 messages

10000 messages

25000 messages

Results

In 5000 messages case all test results were similar. Expect for 6 nodes cluster with 4 shards per queue – performance was Slightly worse. For 10000 messages contrary to 5000 messages case the best performance was recorded on a cluster with 6 nodes and 4 shards. 25000 messages – 2 Nodes cluster was performing better than 6 nodes and the best performance was reached on a cluster with 2 nodes and 6 shards. One producer and five consumers setup testing demonstrated that RabbitMq performs better with smaller amount of nodes if there is need to process more than 1000 messages/second. In case of <1000 messages/second calls bigger cluster performs slightly better. Sharding of the queue increases performance because we had enough consumers to read from each shard and bigger clusters become slower, because messages have to be replicated on each node.

Conclusion

Required cluster setup depends on the load we are putting on a RabbitMQ cluster – bigger cluster do not process messages faster, because replication between nodes slows it down. According documentation RabbitMQ performance do not scale by adding extra nodes it scales by sharding queues and it was proved by sharding our demo queue, more detail on sharding. Best performance was recored when each shard had a consumer if load was higher than 10000 messages.

Add Comment

Your email address will not be published. Required fields are marked *