I am working on converting an existing python based monolith solution to a microservice. The current flow is pretty straight forward:
Accept XLSX as input -> Run some complex algorithms based on input -> generate XMLs
I have created two services using Flask Restplus:
- storage - this deals with store/download/delete of any input or output files. When a user calls /upload/ with an xlsx file, this service will store the given file and return back the download_url to the user back as a response.
- gen - User calls this service by passing the download_url in a request, this service download the file, process it, generate output file, upload the output file to the storage by calling a storage service endpoint.
However, at times, we have seen that the input files we are receiving are quite large in size(~2GB) and it's taking time to upload. Now we are worrying that if multiple users upload huge files concurrently then our system will go for a toss. We have made the gen service async using celery + RabbitMQ. But, I am not sure what needs to be done for the file upload part.