We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected using a self-written PostgreSQL queue lib.

That's how they communicate at the moment:

 1. Call POST request to service A to create a Job in PG Queue
 2. Worker takes a job from Queue and runs calculations
 3. After a successful calculations the worker saves results in the shared DB 
 1. External service B requests every N seconds service A to take the data calculated in the W by job id
 2. Service A observes shared DB and returns the data once the worker is finished (status=Done for a particular job)

old way architecture

We have decided to rewrite the Python service A into Node.JS (NestJS) and improve the current implementation in any way. As you may have noticed, there's line External service B requests every N seconds. We want to get rid of this pattern as well and use something more efficient and performant for this case.

1. Concept using Kafka

The first idea is to use some message queue service like Kafka:


I like this approach since we have a separated worker which only executes some calculations (jobs) and Kafka which plays the role of distributing messages between services. This approach could also get rid of interval/polling requests for taking an actual data.

At the same time it has several possible drawbacks:

  • Imagine the Worker runs only on single thread and, accordingly, can perform one job at a time. How to handle multiple messages at the same time from Kafka in the worker if there is already running job? Do we need to create a RAM Queue or Kafka can handle these cases?
  • What if Kafka or service that consumes messages would suddenly shutdown, will the transmitted data be lost?
  • Maybe use another appropriate MQ such as RabbitMQ instead of Kafka?

2. Concept using PG Queue ... with Kafka

Since we are rewriting the service from scratch using a new technology, we can improve the original approach by replacing the interval requests for getting the data with Kafka. So, the interaction scheme will look like this:

 1. External service B consumes all messages from a particular topic
 2. Service A observes DB and sends data to the topic once the worker is finished (status=Done on the job)

The approach using a shared database and a Postgres queue seems to me more safe and controllable, but at the same time the approach using Kafka as a common message broker seems more modern and sustainable.

Any thoughts on this? Maybe there's better approach to this problem.

2 Answers 2


You have correctly identified a big inefficiency:

External service B requests every N seconds

Any kind of long polling or continuous polling is bound to be inefficient if the normal duration of the job is unknown.

According to your diagram, once the Worker is finished calculating the result, the Worker can do two things:

  1. Persist the results of the work somewhere (a database, an s3 bucket, even Kafka itself with a long retention policy).
  2. Notify others that the work is done (Kafka, AWS SNS, etc).

This will be a much more efficient design that is also not too difficult to implement. You should feel confident to proceed down this path, as long as you are comfortable with learning the lower level details of the Kafka SDK in your language.

  • Thank you for your reply! Okay, I'm now sure we should rewrite the current polling solution with a message queue, but what to do with the database queue?
    – phen0menon
    Jun 17, 2022 at 7:33
  • @phen0menon it might need to go extinct :) you'll have to update the code in such a way that you can gracefully switch over to Kafka. perhaps blue/green deployments would be helpful in this situation. you can confirm everything is working in production before switching over. for the db queue tables, you can also just leave them there and have the application code just not use them anymore.
    – MrUwugu
    Jun 17, 2022 at 13:17

I guess my ideas for a design would depend on the requirements, functional or otherwise.

For example, on how scalable this solution really needs to be. Would a single non-scalable solution actually suffice? It is really easy to write a non-blocking solution these days, capable of handling 10.000s of connections on a single node. If in aggregate a single node with 4-8-16 cores is enough "workers", I would go with that. It's just one application, really easy and simple.

If you need a somewhat scalable amount of workers, you could go with Kafka I guess. The real drawback that I see here, is that Kafka Queues are only as scalable as many partitions you initially create the queue with. If you create the queue with 100 partitions, you'll never be able to scale beyond 100 consumers for that queue. It's because each client will be assigned a partition exclusively. You also would need to make sure work has a random enough key so that they are correctly distributed. Kafka does not distribute based on load nor on client availablility. If your key designates the work packet to an already working client, the packet will just sit there until that client pulls it, even if the other 99 clients are not doing anything.

Also, you can not skip or select messages in Kafka. You can not really wait for a specific message to arrive. So it would be hard to wait for a reply to a specific request for example in Service B. Unless I missed something, Kafka does not really fit your requirements.

A more traditional queue approach would perhaps be better suited. One where you won't have to poll, but be notified when a message of interest arrives.

  • You correctly noticed the scaling issue. Well, I think, it may be scaled horizontally, i.e we can run another worker instances. Will it be a problem to use a single-partition topics? I think since the worker works only on a single thread, it's appropriate solution to achieve jobs ordering; Using another MQ is not particularly suitable solution since we have a specific stack in the team; We don't need to reply to a specific request in the B service. On the contrary, we're trying to avoid this pattern and notify all consumers (services) as soon as the worker completed calculations.
    – phen0menon
    Jun 20, 2022 at 16:06
  • I don't get the notification scheme. Is only a single request allowed in the system? If yes, there's obviously no need for any of this. If not, you'll likely want to identify the job and wait for that particular job to complete, right? Or am I missing something? Jun 20, 2022 at 16:32
  • There's the worker that processes jobs sequentially (i.e only one job can be processed at a time). There's a queue from which worker takes jobs. Once a job is completed, the worker sends a message to the consumers (service B). A message contains full job results. This is pretty simple :)
    – phen0menon
    Jun 20, 2022 at 17:07
  • The problem is that how can we design a queue using Kafka and maybe PG queue
    – phen0menon
    Jun 20, 2022 at 17:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.