What's the best way to do horizontally scalable file uploads?

Question

In general, how do websites handle file uploads when they scale horizontally? Obviously if you have a load balancer going to 10 application servers, and each application server just saves an upload to it's file system, certain files will end up returning a 404 depending on which application server the load balancer pairs you with.

I thought maybe making a TCP server that serves and uploads files to and from a directory, but this both seemed like overkill and this won't work for large files since the app server will need to wait for the entire TCP response to come in before sending it's own response to the browser, resulting in incredibly slow loads for the user.

Another thought I had was proxying the HTTP request to the other app servers if the one originally asked doesn't have it in it's own file system, though I wasn't sure if this would have the same big file problem as a TCP-based file server, or if there was a better, more standard solution.

I don't have a specific situation I need to use this for, I was just thinking conceptually about how this would be handled in an environment that has high scaling demands.

Answer 1

NFS or another filesystem sharing protocol works well here. What you could do is deploy a server dedicated to sharing a high-speed disk array via NFS. Then each server in the load balancer pool would mount that NFS share and write the uploaded file to it. That way, it won't matter which application server handles the upload. Every server will be able to read a file uploaded from any server.

One problem to avoid is file name collision so make sure that you have a mechanism for naming a file uniquely across all of the servers.

Hope this helps.