Using Apache Kafka for rare events [closed]

Question

There are multiple microservices which are responsible for storing data dependent from user, e.g. user preferences on some products, user orders, etc. There is a scenario when user is deleted from user management service's database and the requirement is to also clear that user's data in other services' databases. Using a message broker with publisher (user management service) and subscribers (several microservices interested in such event) seems to be the most intuitive solution for me. I thought of Apache Kafka, but since such events won't occur very often, I have doubts if this "wicked fast" tool, known of its great performance, is a proper one. Could someone comment if it's a good idea? Or maybe should I use other message broker or even other architectural solution?

Answer 1

As SoftwareArchitect123 said, we don't have enough context to say if you should or you shouldn't use Kafka as a solution. I've worked the last year or so pulling in Kafka as a new tech, so I can anecdotally compare them a little bit.

Unless your are handling very large load (billions of records), I would not recommend pulling Kafka in as a new tech (if it already exists in your organization, that's different). Kafka is pretty low-level, and has a much high bar-to-entry than Rabbit or other messaging techs. It can also have a higher maintenance cost (in my anecdotal experience), because of how flexible it is (handling re-balancing, topic-partition offsets, etc.).

If you need to send a few hundred thousand inventory or email requests, I'd recommend a simpler, more consumer-friendly tech like activemq, rabbit, etc.

Answer 2

Due to the limited information available, no clear answer can be given here, but instead of questioning the use of Apache Kafka, it could be useful to search for well-known alternatives.

In such discussions, RabbitMQ is often mentioned as a possible option. The difference between these two technologies has already been extensively debated in Is there any reason to use RabbitMQ over Kafka?.

In a nutshell:

• Even if both technologies implement asynchronous messaging, they were specialized for different application scenarios.
• RabbitMQ is a classic message broker that provides communication between microservices with a low entry hurdle and simultaneously enables complex routing scenarios.
• Apache Kafka on the other hand, specializes in the processing of data streams.

Since your (very vague) description doesn't indicate any data streams, RabbitMQ or another implementation of AMQP (ZeroMQ, ActiveMQ, ...) might be the more appropriate technology. In addition, the first steps are much easier (personal opinion).

I have already used both technologies, but I am much more experienced with RabbitMQ.