1

I am working on converting an existing python based monolith solution to a microservice. The current flow is pretty straight forward:

Accept XLSX as input -> Run some complex algorithms based on input -> generate XMLs

I have created two services using Flask Restplus:

  • storage - this deals with store/download/delete of any input or output files. When a user calls /upload/ with an xlsx file, this service will store the given file and return back the download_url to the user back as a response.
  • gen - User calls this service by passing the download_url in a request, this service download the file, process it, generate output file, upload the output file to the storage by calling a storage service endpoint.

However, at times, we have seen that the input files we are receiving are quite large in size(~2GB) and it's taking time to upload. Now we are worrying that if multiple users upload huge files concurrently then our system will go for a toss. We have made the gen service async using celery + RabbitMQ. But, I am not sure what needs to be done for the file upload part.

Improve this question
2
  • I am curious about this. Here are a couple of links I hit searching. izmailoff.github.io/web/flask-file-streaming stackoverflow.com/questions/44727052/… Not necessarily what you want, but other people's attempts.
    – joshp
    Commented Jan 18, 2020 at 3:04
  • With flask, I wonder do you let the web server completely handle the uploads for you and pass you a reference? Do you stream the input through flask/python to storage. If you read the whole incoming file into memory then you are headed for the failure you fear with a few concurrent users. I'm used to Java.
    – joshp
    Commented Jan 18, 2020 at 3:14

1 Answer 1

Reset to default
2

I could think of these options (Note: not all of them are flask specific).

  1. Cap your file size for upload.
  2. Compress the file before upload. Chunk the file, you can check this.
  3. Stream the file during upload, you will not be able to handle much requests if you keep them all in memory. (as @joshp has mentioned in the comments).
  4. Buff it up, provide more resources when you see your system is reaching capacity. Bring up more instances of your application. Auto scale up and down dynamically.
  5. Back Off/Circuit breaker: When you see your system has reached capacity, do not allow users to temporarily upload, or receive the request and let user know you will start in a while.
  6. Implement Deduplication: If upload is the most frequent operation, worth the effort to implement. Will save network, costs and time.
  7. Reactive framework: If you are relying on threads for concurrency (especially in Python), worth checking out reactive frameworks (RxPy maybe) for upload. This will utilise resource efficiently. Found this implementation called waterbutler if it helps.
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.