Our software has two services - one which provides rest api used by client and one for computations. The computation is quite expansive and can take few hours or days in some cases. There is only one instance of rest-api service and one instance of computation service. Rest-api service communicates directly with computation service and there is always only one computation running.
What we need now is to support more instances of both services to be able to run more computations at the same time. The idea was to use message broker (RabbitMQ or other) and queues. The flow would be
- rest-api publishes request for computation to the queue A
- message broker hands request for computation to one instance of computation service which starts the calculation
- computation services publishes status updates periodically to queue B
- when the calculation is finished the result is published to queue B
- message broker hands results from queue B to one instance of rest-api which store the results
But there are two features which do not fit into this architecture - cancellation of computation and ability to get partial results during calculation.
For cancellation it is necessary - send cancellation request to particular instance of computation service and to get it there immediately if the computation is already in progress - to "remove" request for computation if is is still in the queue
One possible solution could be that the computation service creates its own queue and sends its "identity"(random string) used as routing key with status updates.
- user cancels computation X
- rest-api instance marks the request for computation X in database as cancelled
- computation service which gets the request X will send status update with its "identity"
- rest-api instance which gets status update (it can be different instance than in step 2) will check if request X was cancelled and if so then it takes identity of computation services and uses it as routing key for delivery of cancellation message
- computation service cancels the computation
But this solution is probably not suitable for getting of partial results because user can ask for partial results several times. Any ideas how to implement it within suggested architecture?
Is the architecture suitable for the task?