0

I am trying to assess which of Apache Kafka, RabbitMQ, and Apache Pulsar will meet my throughput requirements for a system with multiple producers and multiple consumers for a single topic.

How can throughput even be compared if systems can be scaled to whatever throughput is needed? In benchmark tests I saw that RabbitMQ has the worst throughput and Kafka and Pulsar are much better (but which is best I've read conflicting things), but I don't understand if these rates like "30 MB/s" are fixed or per-node. It has to be per CPU or per GB or RAM or per queue or something, and I'm confused which systems can be easily scaled as needed.

I read RabbitMQ queues are bound to one node. But for Quorum queues they are replicated. I'm guessing this type of replication helps with data safety but not throughput? Is there really a fixed total throughput RabbitMQ supports, and if you need to go beyond that then you're out of luck?

I read Kafka has high throughput from a lot of optimizations like sequential disk reads, but does this just imply a fixed higher amount of throughput or can it be scaled to infinity?

Any significant differences between Kafka and Pulsar, or is one just a fixed ratio more efficient?

Basically wondering what resource limits the total possible throughput of a single queue/topic in each system, e.g. disk, number of nodes, or whatever.

1

[...] if systems can be scaled to whatever throughput is needed?

this is a false assumption, be careful.

RabbitMQ is message oriented, so the base assumptions, specially the Exactly-Once guarantees usually makes it less performable.

Kafka on another hand only supports a single reader per consumer group per topic per partition. So, your biggest bottleneck is there. You usually can design around this but not to infinity and beyond.
If you need event ordering though, you can't design around, this is a hard stop.
If you need Exactly-Once guarantees, you will also loose performance.

I cant produce specific numbers for such comparisons but for general use cases Kafka performs better. But if your constraint Kafka forcing it to operate like a message queue, it will probably loose to RabbitMQ...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.