I have 2 microservices [A], and [B]. These microservices run in a load balanced environment and there may be N instances if each microservice running. The microservices communicate via a message framework (Rabbit MQ).
[A] Starts a long running process loop (P) which modifies data.
[B] Can send cancellation requests to [A] (triggered by user input). This is so that the long running process in [A] can be stopped before completion. [A] will record the cancellation request in a database table (a form of distributed lock, because N instances of [A] could be running and so the instance that receives the message may not be the instance that is running the processing loop (P).
[A] checks for cancellation requests in each iteration of a loop. Where a cancellation request is found, processing is stopped and data modifications are rolled back.
The difficult part is this:
I also need to account for the possibility of [A] falling over. Where it does, the iterative check for the cancellation request will not occur because the processing loop is no longer running, but data modifications still need to be rolled back.
I am looking at splitting the tasks of stopping the processing loop (P) and rolling back the data modifications. The thread on [A] that handles the cancellation request message receipt could perform the rollback, but it needs to know that the processing loop (P) has REALLY stopped. Given that the processing loop (P) and message receipt could occur on different instances of [A], I'm unsure how to architect this.
Currently I am setting up [A] to record processing progress, so I can check at the point of message receipt to see if processing has not occurred for a period of time that would suggest that processing loop (P) has failed. But this feels less than robust as (P) could just be hanging or slow.
Is there a precedent or common solution for this type of issue?
How can I ensure that the data modification rollback occurs whatever happens?