0

In general, how do websites handle file uploads when they scale horizontally? Obviously if you have a load balancer going to 10 application servers, and each application server just saves an upload to it's file system, certain files will end up returning a 404 depending on which application server the load balancer pairs you with.

I thought maybe making a TCP server that serves and uploads files to and from a directory, but this both seemed like overkill and this won't work for large files since the app server will need to wait for the entire TCP response to come in before sending it's own response to the browser, resulting in incredibly slow loads for the user.

Another thought I had was proxying the HTTP request to the other app servers if the one originally asked doesn't have it in it's own file system, though I wasn't sure if this would have the same big file problem as a TCP-based file server, or if there was a better, more standard solution.

I don't have a specific situation I need to use this for, I was just thinking conceptually about how this would be handled in an environment that has high scaling demands.

  • Why will the app server need to wait for the entire TCP response to come in before sending its own response to the browser? – user253751 Apr 26 '17 at 4:39
  • Well specifically I'm working with Python and I couldn't think of a way to get a not fully-loaded TCP buffer and pass it as a response to, say flask, or really any other web framework – Danny M Apr 26 '17 at 4:42
  • 1
    I would say this question is way too broad, as the problem domain is too complex to properly answer in Q&A format. Also, there seems to be two questions here, first asks about general architecture of uploading files to multiple servers, and second is about concrete implementation of file transfer in Python. – Euphoric Apr 26 '17 at 5:55
  • this post is rather hard to read (wall of text). Would you mind editing it into a better shape? – gnat Apr 26 '17 at 6:49
3

NFS or another filesystem sharing protocol works well here. What you could do is deploy a server dedicated to sharing a high-speed disk array via NFS. Then each server in the load balancer pool would mount that NFS share and write the uploaded file to it. That way, it won't matter which application server handles the upload. Every server will be able to read a file uploaded from any server.

One problem to avoid is file name collision so make sure that you have a mechanism for naming a file uniquely across all of the servers.

Hope this helps.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.