1

In a multi server system, e.g. a load balancer, multiple web servers and a database server, where do you store the files / images users upload when each web server will need access?

Before now I've just used a single web server, or maybe one web server and a MySQL database server. But if you have multiple web servers I guess you have a range of options for making sure resources like images and other user files are shared, such as:

  • Storing them on the database server
  • Or a server specifically for them
  • Maybe mounting the same directory
  • Or using a tool like rsync to synchronize the user uploads directories on each web server

Or maybe you have a different solution? or there's a de-facto way of doing this that I'm not aware of?

  • You might also use something like Amazon S3 – GrandmasterB Feb 18 '14 at 18:13
  • Hi GrandmasterB, thanks I've looked into Amazon S3 a bit just now, my only concern I guess is the cost is difficult to work out atm without working out exactly how much we will use, get/put requests and estimated size of the images, but I'll see if there's any way i can work out these estimates easily. – stilliard Feb 18 '14 at 18:44
3

I would consider a few options, depending on several factors:

  • If it's not a huge volume of images, then a single fileserver accessible by all the webservers is the easiest route.

  • If the images are very small (much less than a megabyte...), a Key-value storage can be considered. (but note that a file system is a great and efficient key-value when the values are more than a few kilobytes)

  • If the content is read much more than modified (at least 100 or 1000 times more frequently), then replicated (via rsync or unison) can reduce lag time. But this advantage is unlikely to be significant. A fileserver is still a great option.

  • If the amount of images is more than what's convenient to store on a single server, then a distributed storage system allows for near-infinite growth. (ceph RADOS, moosefs, glusterFS, etc.)

  • Hi Javier, thanks I'm definitely leaning to a separate file server now for this, for now we shouldn't have too many images, several thousand but not enough for a distributed storage system atm. With this setup do you recommend mounting the file server in each web server to a specific uploads directory, or uploading to it directly from the application, or another method? – stilliard Feb 18 '14 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.