0

So I'm trying to create a simple website where I run a computationally intensive task on the backend based on input submitted by a form. For reference: my goal is not too dissimilar from something like this. Here the underlying computation that's being performed is too intensive to be run by the webserver and should be handled separately by a worker. I currently have the worker that performs the computation written in Python. My current plan for the architecture is to run microservices in Docker containers. There would be a container for the web engine (nginx), the web app (flask), and the math computation (this has already been written in python).

My question is: What is the best way for me to communicate between my web app and my computation service? My current thought is to use a message queue (probably RabbitMQ) deployed as a separate container with a callback. It would function pretty similarly to this example/tutorial. When I make the RPC I would add the outstanding web request to a hashmap that maps from a unique queue identifier to the outstanding web request. That way when I pop the response from the callback queue, I know which web request to send that answer to.

Firstly, is this the best approach? Am I missing a better and/or more natural way to accomplish this? I've considered just exposing my worker computation via a RESTful api designed in flask. I'm not sure if this is better/worse.

Secondly, how do i handle RPC calls that I never get a response for? If something were to go wrong and a response never gets placed back into my callback queue, the original web request would sit in my hashmap indefinitely. Is there a common design pattern here that's used to prevent this?

Improve this question
1
  • Are you keeping the outstanding web request alive all the time? Regarding the hashmap, you are indexing the request but looks like you need caching. Have you considered implementing a cache? Something like Ehcache or similar? Finally, how intense are these tasks? What's the average response time?
    – Laiv
    Commented Jun 16, 2020 at 7:28

1 Answer 1

Reset to default
1

Whether this is the best approach depends on many unknowns, but it's one reasonable approach, and I would simply start with it. If you find deficiencies or changes in requirements later you may have to redesign but that's exactly what all the big players do from time to time.

Regarding the second question, you can use timeouts for simplicity, or process monitoring in the worker container for accuracy if you have a way of periodically checking whether each outstanding request is being worked on.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.