Images are uploaded to the server once in a while. Users send an API request for downloading all of those images that were uploaded to the server (Images reside in the server itself). Instead of downloading every single file, the server would collect all the images, archive it and then send the zip file to the user.
Problem with this approach is that, even for a single user, the CPU usage is very high during the archive process, but the server needs to support more than 100 concurrent requests for the download. Also, images could be uploaded at any moment to the server, so pre archived files are also not a solution.
One possible solution for my use case: Since the images are the same for all users, the server would archive all the images in any of the first requests, then on the following requests, check if any new image has been added (maybe store the last JSON in memory, and diff it with the current one). If new images are not added, send the previously archived zip file, else, archive it again. But this would still be the same if different users have different files to download.
What could be done in this scenario? Thoughts? Solutions?