1

I have two types of microservices: Data Collector and Data Processor. They communicate with each other via Azure Service Bus Queue. Data Collector collects data from different sources and, once done, sends a "start processing" message to the queue. An instance of Data Processor receives the message and starts data processing.

enter image description here

If the instance of Data Processor dies during processing, another instance should start the processing again.

My question is, how to let this "another" Data Processor instance know that it should start the processing the previous instance wasn't able to finish?

The best approach I have found is message locking. When the Data Processor receives the "start processing" message, it does not delete it from the queue, just locks it, so that nobody else would start processing the same message. Once processing is done, it deletes the message from the queue.

The problem is that processing can take up to 2 hours, but Azure Service Bus only alows locks up to 5 minutes. There is a way to configure lock renewal, but that looks like a hack already, mostly because the renewal is not guaranteed.

Another option is to make Data Processor store the message somewhere in its database so that other instances of Data Processor can periodically check the status of the message and start its processing. But that seems even worse than locking, because other instances will have to periodically poll the database for unprocessed messages.

Is there another way to ensure the data processing will be finished eventually, or message locking is a good fit for my case?

Thank you!

Improve this question
3
  • One way is to use a dead-letter queue. Check this page for some ways to implement this pattern.
    – ByteArtisan
    Aug 17, 2021 at 22:45
  • @ByteArtisan, thank you for the suggestion! I have read the article. Not sure the dead letter queue could help here though: it is applicable when the consumer cannot handle the message because the external service is unavailable, while I'm trying to handle a situation when the consumer itself crashes during message processing.
    – yaskovdev
    Aug 18, 2021 at 10:14
  • 1
    In that case maybe the service bus PeekLock is what you are looking for. If the worker crashes, the message it will go automatically to dead-letter after the specified timeout. Still you have to pool dead letter messages and find a way to avoid continues loop of the same message. I general, you can catch any error to check if it is transient error or not and decide if should go to dead letter queue. PeekLock feature is only need to cover some machine power failure or process crash.
    – ByteArtisan
    Aug 18, 2021 at 10:35

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.