Is Kafka a good idea for connecting a REST API with Worker which runs large computations per job in a queue?

Question

We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected using a self-written PostgreSQL queue lib.

That's how they communicate at the moment:

 1. Call POST request to service A to create a Job in PG Queue
 2. Worker takes a job from Queue and runs calculations
 3. After a successful calculations the worker saves results in the shared DB 
- 
 1. External service B requests every N seconds service A to take the data calculated in the W by job id
 2. Service A observes shared DB and returns the data once the worker is finished (status=Done for a particular job)

We have decided to rewrite the Python service A into Node.JS (NestJS) and improve the current implementation in any way. As you may have noticed, there's line External service B requests every N seconds. We want to get rid of this pattern as well and use something more efficient and performant for this case.

1. Concept using Kafka

The first idea is to use some message queue service like Kafka:

I like this approach since we have a separated worker which only executes some calculations (jobs) and Kafka which plays the role of distributing messages between services. This approach could also get rid of interval/polling requests for taking an actual data.

At the same time it has several possible drawbacks:

• Imagine the Worker runs only on single thread and, accordingly, can perform one job at a time. How to handle multiple messages at the same time from Kafka in the worker if there is already running job? Do we need to create a RAM Queue or Kafka can handle these cases?
• What if Kafka or service that consumes messages would suddenly shutdown, will the transmitted data be lost?
• Maybe use another appropriate MQ such as RabbitMQ instead of Kafka?

2. Concept using PG Queue ... with Kafka

Since we are rewriting the service from scratch using a new technology, we can improve the original approach by replacing the interval requests for getting the data with Kafka. So, the interaction scheme will look like this:

 1. External service B consumes all messages from a particular topic
 2. Service A observes DB and sends data to the topic once the worker is finished (status=Done on the job)

---

The approach using a shared database and a Postgres queue seems to me more safe and controllable, but at the same time the approach using Kafka as a common message broker seems more modern and sustainable.

Any thoughts on this? Maybe there's better approach to this problem.

Answer 1

You have correctly identified a big inefficiency:

External service B requests every N seconds

Any kind of long polling or continuous polling is bound to be inefficient if the normal duration of the job is unknown.

According to your diagram, once the Worker is finished calculating the result, the Worker can do two things:

1. Persist the results of the work somewhere (a database, an s3 bucket, even Kafka itself with a long retention policy).
2. Notify others that the work is done (Kafka, AWS SNS, etc).

This will be a much more efficient design that is also not too difficult to implement. You should feel confident to proceed down this path, as long as you are comfortable with learning the lower level details of the Kafka SDK in your language.

Answer 2

I guess my ideas for a design would depend on the requirements, functional or otherwise.

For example, on how scalable this solution really needs to be. Would a single non-scalable solution actually suffice? It is really easy to write a non-blocking solution these days, capable of handling 10.000s of connections on a single node. If in aggregate a single node with 4-8-16 cores is enough "workers", I would go with that. It's just one application, really easy and simple.

If you need a somewhat scalable amount of workers, you could go with Kafka I guess. The real drawback that I see here, is that Kafka Queues are only as scalable as many partitions you initially create the queue with. If you create the queue with 100 partitions, you'll never be able to scale beyond 100 consumers for that queue. It's because each client will be assigned a partition exclusively. You also would need to make sure work has a random enough key so that they are correctly distributed. Kafka does not distribute based on load nor on client availablility. If your key designates the work packet to an already working client, the packet will just sit there until that client pulls it, even if the other 99 clients are not doing anything.

Also, you can not skip or select messages in Kafka. You can not really wait for a specific message to arrive. So it would be hard to wait for a reply to a specific request for example in Service B. Unless I missed something, Kafka does not really fit your requirements.

A more traditional queue approach would perhaps be better suited. One where you won't have to poll, but be notified when a message of interest arrives.