Architecture for downloading large number of large images

Question

I am running a website for a customer who has terrabytes worth of images. Each JPG image is high-res (20MB) and belongs to a hierarchy like this:

Group A
    SubGroup 1
        subsubgroup a
            Image A1a.1
            Image A1a.2
            ...

Currently the customer can only view images for any given subsubgroup at any one time, meaning only ~30 images get loaded on the page. Not a problem.

The images are stored in a cloud bucket (GCP). Currently, my server just sends the pre-signed URLs and the client loads them.

The customer has request the ability to mass download all images in a Group from their browser.

Some Groups can contain hundreds of SubGroups, and each one of these can contain multiple subsubgroups. As an illustrative example, for a Group of 20 SubGroups, we're talking about 20GB of data.

How can I achieve this reliably?

Possible solutions:

1. Download all images on the server, create a zip (or number of zips) and upload to bucket. Share link with customer. Would probably need to run as a background job. This feels very flaky and I am doubtful it'll even work given the size of the data.

2. Create a new page that renders all images. My server only sends the pre-signed URLs from GCP bucket. Add some basic JS functionality to download all images on the page.

3. Some sort of SFTP server?

---

My current stack is relatively minimal:

• 1 cloud VM
• Flask app, uwsgi server, behind an NGINX reverse proxy
• Cloud storage (GCP) for images
• MySQL database for image hierarchies

Answer 1

If the requirement is really to download a huge number of images and no just scroll through them, then the ZIP solution would probably work well. I've recently had the need to download a huge number of vacation photos from my Google account, and they offer exactly this as a mass download option.

Creating the ZIP files should actually not be too time-consuming, as you would not need to compress the files, just copy them to the archive (JPEG images are well compressed already).

You need to manage your user's expectations, though. Downloading 4GB over a reliable broadband connection is pretty OK, 20GB over an ADSL link will likely not make them happy. If you're really generous, you can offer a set of ZIP files with some size limit so they can download the data piecewise.

Answer 2

If the images are already stored in a GCS bucket, you should leverage the fact that cloud storage solutions are already very good at exactly what you want to do, which is transfer large amounts of data reliably.

If the customer needs to download a large number of huge files, use existing tools such as gsutil (GCS's command line tool) to download them. It should be hard to build the appropriate command line to recursively download a Group from GCS and present it to the customer, instructing them to run it locally.

Don't reinvent the wheel. GCS has a supported tool for reliable, resumable file downloads. Use it.

Answer 3

You might be overthinking this or, rather, creating a challenge that doesn't really exist, if you think about this a little differently.

I would suggest that you think about this not as a 'mass' download, but as a local, persistent disk-based cache of the images. Your groups of (sub)groups has the same structure as folders in a file system. The general idea here would be to create a local user directory which your interface creates and manages.

When the user views a group (or sub-group), your interface first checks the local path that matches your group/subgroup structure. If a file is found locally, you can display that. If it not, you pull from GCP.

The last part of this would be how the local disk store is populated with the actual images. You could provide options to save or cache individual files, or leaf groups upon viewing, or you could recursively save groups/subgroups. This avoids the challenges around downloading very large files because you are simply pulling the images individually. And really, there's not much advantage to zipping because the images won't compress significantly, and you basically need the full file to get anything out of it. If you fail to get an image, you only miss that one image and you can build a retry process for those failures, e.g.: re-add the URL to the end of the download list.

The biggest challenge here, I think, is dealing with local file access and directory management from a browser-based JS solution. Seems like there's been some improvement in the capabilities required for that kind of solution over time, but it's not really my area of expertise.