architecture for services doing very expensive computations

Question

Our software has two services - one which provides rest api used by client and one for computations. The computation is quite expansive and can take few hours or days in some cases. There is only one instance of rest-api service and one instance of computation service. Rest-api service communicates directly with computation service and there is always only one computation running.

What we need now is to support more instances of both services to be able to run more computations at the same time. The idea was to use message broker (RabbitMQ or other) and queues. The flow would be

1. rest-api publishes request for computation to the queue A
2. message broker hands request for computation to one instance of computation service which starts the calculation
3. computation services publishes status updates periodically to queue B
4. when the calculation is finished the result is published to queue B
5. message broker hands results from queue B to one instance of rest-api which store the results

But there are two features which do not fit into this architecture - cancellation of computation and ability to get partial results during calculation.

For cancellation it is necessary - send cancellation request to particular instance of computation service and to get it there immediately if the computation is already in progress - to "remove" request for computation if is is still in the queue

One possible solution could be that the computation service creates its own queue and sends its "identity"(random string) used as routing key with status updates.

1. user cancels computation X
2. rest-api instance marks the request for computation X in database as cancelled
3. computation service which gets the request X will send status update with its "identity"
4. rest-api instance which gets status update (it can be different instance than in step 2) will check if request X was cancelled and if so then it takes identity of computation services and uses it as routing key for delivery of cancellation message
5. computation service cancels the computation

But this solution is probably not suitable for getting of partial results because user can ask for partial results several times. Any ideas how to implement it within suggested architecture?

Is the architecture suitable for the task?

Answer 1

It is suitable and correct.

I think the thing you are missing is the concept of 'fan out routing'

So you have a single queue of computation requests which are picked up and taken off as computation workers become available. This queue subscribes to the RequestComputation message and all the workers pull from it.

But you also have each computation worker create its own queue which subscribes to the cancelComputation and requestPartialResult messages.

Thus when the REST api publishes one of those messages, it 'fans out' and will be picked up by all the computation workers. Each of which can check to see what job its working on and respond if required.