I have a system which has 5 services. Each are run in a separate process. They resemble an assembly line, in the sense that they mostly process messages in the same order 1->2->3->4->5, however it's not always the case that the processing is linear. There are cases where it can skip though the chain (e.g. 1->2->5) or go backwards (e.g. 1->2->4->3->1) or even zig-zag (1->2->4->3->5).
The system is fed payloads to process one by one, starting with service 1. Given the system is asynchronous and non-linear, I cannot really know when the whole system processed each of the payloads. On the other hand, I want the system to stop sometimes and the best choice is as soon as it processed all the payloads.
These solutions would not be too good:
- A simple kill or using a poison-pill approach would not work (example https://stackoverflow.com/questions/13847963/akka-kill-vs-stop-vs-poison-pill), as then I could possibly end up not processing the payload completely (1->2->1 would fail, as 1 would have been stopped after 2 received the last payload)
- I can do a timeout approach, however that would not work for two reasons: a) it will hinder development / testing (e.g. if I waited 1 minute for a 5-sec payload effective processing time), b) timeouts still can be too low, so I could run into issues if some services did not finishe processing
I was thinking about creating another service that would serve as a two-phase commit kind of stopper. The idea would go along these lines:
- Any service could call stopper to say "I want the system to stop"
- Stopper would call all other services and tell them "Stop has been requested"
- Other services will wait till they process what they have in the queue and tell stopper "I can to be stopped"
- Stopper will wait till it collects "I can be stopped" from all services, then broadcast "Perform stop"
- All services will then go ahead and stop their respective processes in a graceful manner
I have issues with the above:
- It looks overly complex for the benefit (compared to "just wait 1 minute then stop" and not adding any non-business code), not sure if worth the trade-off
- I would have to infuse logic related to stopping into all services and it would have to be coupled to both the queuing mechanism (so it knows when there are not more payload messages) and business logic (so it knows when the payload itself is done processing)
- There are still some edge cases to be considered. E.g. in 1->2->1 scenario, 1 could finish, 2 could be processing the last payload, when it's done it could send a message to 1 and to stopper service, stopper will send "Perform stop" and 1 could receive that message before it received a message from 2 that it should process, so it could stop prematurely
Are there best practices or examples of how to achieve this in some other way? Or maybe constraining the system in some other way that would make this problem disappear / easier to manage?