So I'm trying to create a simple website where I run a computationally intensive task on the backend based on input submitted by a form. For reference: my goal is not too dissimilar from something like this. Here the underlying computation that's being performed is too intensive to be run by the webserver and should be handled separately by a worker. I currently have the worker that performs the computation written in Python. My current plan for the architecture is to run microservices in Docker containers. There would be a container for the web engine (nginx), the web app (flask), and the math computation (this has already been written in python).
My question is: What is the best way for me to communicate between my web app and my computation service? My current thought is to use a message queue (probably RabbitMQ) deployed as a separate container with a callback. It would function pretty similarly to this example/tutorial. When I make the RPC I would add the outstanding web request to a hashmap that maps from a unique queue identifier to the outstanding web request. That way when I pop the response from the callback queue, I know which web request to send that answer to.
Firstly, is this the best approach? Am I missing a better and/or more natural way to accomplish this? I've considered just exposing my worker computation via a RESTful api designed in flask. I'm not sure if this is better/worse.
Secondly, how do i handle RPC calls that I never get a response for? If something were to go wrong and a response never gets placed back into my callback queue, the original web request would sit in my hashmap indefinitely. Is there a common design pattern here that's used to prevent this?