Cancelling a background service from another microservice in load balanced environment

Question

I have 2 microservices [A], and [B]. These microservices run in a load balanced environment and there may be N instances if each microservice running. The microservices communicate via a message framework (Rabbit MQ).

[A] Starts a long running process loop (P) which modifies data.

[B] Can send cancellation requests to [A] (triggered by user input). This is so that the long running process in [A] can be stopped before completion. [A] will record the cancellation request in a database table (a form of distributed lock, because N instances of [A] could be running and so the instance that receives the message may not be the instance that is running the processing loop (P).

[A] checks for cancellation requests in each iteration of a loop. Where a cancellation request is found, processing is stopped and data modifications are rolled back.

The difficult part is this:

I also need to account for the possibility of [A] falling over. Where it does, the iterative check for the cancellation request will not occur because the processing loop is no longer running, but data modifications still need to be rolled back.

I am looking at splitting the tasks of stopping the processing loop (P) and rolling back the data modifications. The thread on [A] that handles the cancellation request message receipt could perform the rollback, but it needs to know that the processing loop (P) has REALLY stopped. Given that the processing loop (P) and message receipt could occur on different instances of [A], I'm unsure how to architect this.

Currently I am setting up [A] to record processing progress, so I can check at the point of message receipt to see if processing has not occurred for a period of time that would suggest that processing loop (P) has failed. But this feels less than robust as (P) could just be hanging or slow.

Question:

Is there a precedent or common solution for this type of issue?

How can I ensure that the data modification rollback occurs whatever happens?

Answer 1

Currently I am setting up [A] to record processing progress, so I can check at the point of message receipt to see if processing has not occurred for a period of time that would suggest that processing loop (P) has failed. But this feels less than robust as (P) could just be hanging or slow.

Let me explain in brief how this thing worked in a well-known distributed system Apache Kafka.

1. Detecting machine failures - session.timeout.ms

   Sending heart-beats to detect the complete machine failures.

2. Detecting process failures - max.poll.interval.ms

   Based on the configurable amount of time, if the process isn't complete, the process is considered choked and retried.

Refer : This Link

Based on #1 & #2 and my understanding, identifying choked processes may be tricky and the most generic way is to make them time driven. Tracking the process-ids & killing the halted/choked process (using process-ids) is another thing that can be done (if possible in the programming language) before rolling the state back.

Answer 2

What I have settled on so far is this.

As per @Sahil Gupta's response, the only way to manage this seems to be setting configurable timeouts after which a process can be considered 'choked'.

In the workflow I described, what we can guarantee, is that once the cancellation has been requested, we can guarantee that it will at some point be processed. The simplest and most reliable place to request the rollback of data is in the same process that received the cancellation message. This process flags the cancellation, and then waits for confirmation that processing has stopped, or a timeout - whichever comes sooner. That same process can then proceed with the rollback of modified data.

I would be very interested to hear if anyone knows of anyway to improve on this further than relying on timeouts.