1

Would appreciate any recommendations regarding design a high-load image processing service.

There are lots (tens of hundreds of relatively large images (~10...20 Mb each)) that to be processed using command line utils when requested.

Generally: request->image processing->result. Obviously, the system must be able to scale to accommodate considerable load.

My concern and I believe the core of the system design a file storage:

  • Using one single network file storage (file share, samba or something like this) seems ineffective, as this one and only network storage is a bottleneck.

  • Having multiple instances of the same file storages on each node (and somehow replicate files) is quire expensive.

  • Storing images in db? For me sounds virtually unusable for large files and command line processing.

I considered more complex designs like having one large storage and cache frequently used files locally; but it seems a bit too complicated and not reliable.

Are there any best practices or patterns for problem like this one?

Improve this question
12
  • 4
    You'll probably need to define what "considerable load" means. For some people that means one image per minute. For others it might mean a hundred thousand images per second. Your architecture is likely to depend on the actual load. You probably also need to get at least some idea of the time to process a single image--governs how many boxes you need to do processing (probably CPU bound) vs. file serving (probably I/O bound).
    – Jerry Coffin
    Nov 14, 2017 at 19:05
  • 1
    Note that “tens of hundreds” of 20MB images may be much less than 1TB, which is well available with cheap hard drives. With a suitable RAID level (e.g. mirroring) the throughput of a disk array can be scaled. So it's not clear that a network storage would present a bottleneck, or that replication would be unaffordable. You really need to estimate how many images you will be loading from disk per second during peak times, and work out a solution from there that fits your budget.
    – amon
    Nov 14, 2017 at 19:20
  • 2
    Possible duplicate of How best to store images on a file system
    – gnat
    Nov 14, 2017 at 19:36
  • 2
    see also: Store and Tag Millions of Images
    – gnat
    Nov 14, 2017 at 19:36
  • 3
    The typical design for a picture archiving and communication system (PACS) is to have a database that keeps records of metadata about the images, including where they are located on the filesystem, but to store the images as flat files on the filesystem. You do NOT want to keep the images in the database as BLOBs. The filesystem itself is usually a logical volume on a SAN.
    – Matt
    Nov 14, 2017 at 20:35

2 Answers 2

Reset to default
3

You basically have two "directions" in which you can scale file system storage: vertically and horizontally.

Vertical scaling is basically just making a drive faster. The obvious move here is from single spinning disc to SSD, possibly in a RAID. This lets you create a file system that has higher bandwidth and throughput. Back before SSDs were common, you might have considered using a number of actual hard drives, but in a RAID to get more bandwidth out of what shows up to the user as a single logical drive. This can still be worth considering if you need more storage than you're willing to buy SSDs for.

Horizontal scaling in this case is a matter of having a number of separate "file systems" that look like a single file system. Fortunately you're dealing with files that are individually small enough to plan on just storing each on an individual file system (unlike a database that might need a single file of a hundred terabytes or more). This means you can scale horizontally fairly easily by (for example) having a number of servers that each store a specified subset of the files you care about, and use (for example) a hash of the file name to determine which server will store each file.

So, to use this you'd basically have a number of file servers, and a list of files to process. If you're going to have multiple clients processing the files, split the list between them. Each client goes through the names in its part of the list, hashes the name to find the server for that file, and retrieves/processes the file.

Depending on the network and what you're doing with the result of the processing, there's another possibility to consider: start with the same set of servers and the same input list. Hash the names, and send each name directly to the server where that file is stored. Instead of retrieving the file and sending it over the network, the server retrieves and processes the file itself.

The latter can have a significant advantage if the result of processing should be stored together with the original file and/or you may have to deal with a relatively slow network (since it only requires transmitting the file names over the network, which we normally expect to be a lot smaller than the file contents).

Improve this answer
4
  • 1
    SSD's don't just "appear" to be faster. They actually are faster.
    – Robert Harvey
    Nov 15, 2017 at 0:41
  • 1
    @RobertHarvey: Yes, the "appear to be" was referring more to using RAID to produce what appears to be a single fast drive out of a number of slower drives. I've reworded to try to clarify.
    – Jerry Coffin
    Nov 15, 2017 at 0:43
  • Thank for the answer, Jerry. I need to consider the approach you propose ( having subsets of files on individual nodes). Generally, it seems it is not scaling, it's more splitting one system to many. Here I can't add or remove nodes arbitrarily when necessary, and in general, adding or remove nodes needs lots of work to relocate/distribute files between nodes. Anyway, thank you!
    – Dmitry Karpezo
    Nov 15, 2017 at 9:53
  • 1
    @DmitryKarpezo When distributing files across nodes, you typically use consistent hashing so that as nodes are added or removed, they are gradually redistributed among themselves. Note that no matter how you design it, there must be some redundancy (each file is stored as at least two, maybe three or more copies), otherwise it is impossible to allow nodes to be removed without losing access to some files.
    – rwong
    Nov 16, 2017 at 4:54
2

You can utilize a distributed file system such as GFS or HDFS or use a storage area network which will increase your architecture complexity but will allow you to scale horizontally a little bit better.

Your bottleneck will be the network IO at this point.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.