0

Scenario:

Images are uploaded to the server once in a while. Users send an API request for downloading all of those images that were uploaded to the server (Images reside in the server itself). Instead of downloading every single file, the server would collect all the images, archive it and then send the zip file to the user.

Problem:

Problem with this approach is that, even for a single user, the CPU usage is very high during the archive process, but the server needs to support more than 100 concurrent requests for the download. Also, images could be uploaded at any moment to the server, so pre archived files are also not a solution.

Possible Solution

One possible solution for my use case: Since the images are the same for all users, the server would archive all the images in any of the first requests, then on the following requests, check if any new image has been added (maybe store the last JSON in memory, and diff it with the current one). If new images are not added, send the previously archived zip file, else, archive it again. But this would still be the same if different users have different files to download.

What could be done in this scenario? Thoughts? Solutions?

Thank you.

| improve this question | |
  • Add the new images to the old zip file? – Johannes Kuhn Jan 27 '18 at 18:35
  • On each upload request? – Pratish Shrestha Jan 28 '18 at 6:47
2

Let the users figure this out.

What you can't handle is a user downloading every single file individually or making you archive every single file whenever they please.

So don't. Once a day update the archive and provide a link to that as well as a link to every image uploaded today. Let the user decide if they really need the old stuff, todays, or both.

| improve this answer | |
  • Oh nice, so basically the server would run a cron that would archive the images at the same time each day. When the user requests for the images, the server would query the database for images that have been uploaded after the time of the cron. And send the URL for both the archived file and the new images added in a JSON response. – Pratish Shrestha Jan 28 '18 at 6:41
  • Just curious, how do services like google drive archive files for each individual user. The user gets to select each file they archive and there must be thousands of users doing this concurrently. – Pratish Shrestha Jan 28 '18 at 6:44
  • Sure, a cron would be perfect for this. You can even use it to update the download page. – candied_orange Jan 28 '18 at 6:45
  • You'd have to ask google. – candied_orange Jan 28 '18 at 6:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.