I have server A that received millions of requests per day and for every request it notifies server B about it. sometimes server B is down (maintenance, bugs etc..) so requests from server A got lost. (both servers are written in nodejs)

I thought to add a queue (Kafka/RabbitMQ) in server A so every time a request failed it will get into the queue and the service will retry to send it again and again every X minutes.

Do you think queue is the right solution? or is there other mechanism that I should consider?

  • If you are considering a queueing architecture, it’s worth considering if B should pull from the queue instead of having a service retry pushes – gntskn Jul 19 at 18:41
  • the requests from a to b should be very fast because once a request get to a then b should be notified because all ecosystem starts using b and I'm afraid that b queuing from a will not be that fast. is my perception is wrong? – Avi L Jul 19 at 18:47
  • 2
    Kafka is very suitable for your needs. If you don't need the notification to be synchronous-like, instead of A notifying B directly, just add a record to Kafka's topic and let B consume from it. That way, even when B goes down and then goes back up, it can simply continue from the offset where it left before going down. – Andy Jul 19 at 19:58
  • I don't know enough about queues to know how the performance compares to a plain HTTP request, but my advice would be to make the system simple and then worry about performance optimization when you know it matters. – gntskn Jul 20 at 16:48
  • @AviL As always it depends. What does it mean in your case Notify? What if server A processes a request but unable to put the notification into the queue? Is it a problem or not? Or in the other way around server A fails to process the request successfully, but notification has already been placed on the queue. Is it a problem or not? If these two actions should be treated atomically then Outbox pattern can be your friend. – Peter Csala Jul 21 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.