I have two types of microservices:
Data Collector and
Data Processor. They communicate with each other via Azure Service Bus Queue.
Data Collector collects data from different sources and, once done, sends a "start processing" message to the queue. An instance of
Data Processor receives the message and starts data processing.
If the instance of
Data Processor dies during processing, another instance should start the processing again.
My question is, how to let this "another"
Data Processor instance know that it should start the processing the previous instance wasn't able to finish?
The best approach I have found is message locking. When the
Data Processor receives the "start processing" message, it does not delete it from the queue, just locks it, so that nobody else would start processing the same message. Once processing is done, it deletes the message from the queue.
The problem is that processing can take up to 2 hours, but Azure Service Bus only alows locks up to 5 minutes. There is a way to configure lock renewal, but that looks like a hack already, mostly because the renewal is not guaranteed.
Another option is to make
Data Processor store the message somewhere in its database so that other instances of
Data Processor can periodically check the status of the message and start its processing. But that seems even worse than locking, because other instances will have to periodically poll the database for unprocessed messages.
Is there another way to ensure the data processing will be finished eventually, or message locking is a good fit for my case?