Flexible image REST api with cdn and caching

Question

Consider a social network in which a user photo is needed in different resolutions large/small/medium etc.

When the photo is uploaded, I do not want to make the user to wait too long for the image to be processed and the different urls for each size to be returned.

I want just to return the base url ex domain.com/images/something.jpg Then when the frontend calls for domain.com/images/something.jpg?s=large I generate the large image, cache it somwhere and always return that.

The problem that I cannot figure out with the above approach is that I cannot use s3 anymore without double transferring the data.

With the above aproach first I need to transfer the data from s3 -> backend server then from backend server -> frontend.

If I go withe the solution that I do not like, frontend waits longer for backend to generate all image sizes and return all urls, then only one transfer need to be done, from s3 to frontend.

I also thought of another possible solution that backend returns urls withhout the image being avaible, for example:

[
  large: domain.com/dd.jpg?s=large
  small: domain.com/dd.jpg?s=small
]

Then when the frontend calls for example the small url, if it does not exist first is generated, then stored in s3 then backend returns 302 redirect to the s3 url. I think that this approach will be bad because of many 302 redirects, but offers the flexibility that I want.

I am still trying to figure out how to solve the problem. Any suggestions please ?

Answer 1

If you don't want the users to wait on upload, you probably don't want them to wait when requesting the specific size of an image either (since in general, the user who uploaded the image would be the first one who will request it anyway). In this case, there is a third solution: generate the scaled versions of an image through a cron job or a background task.

When the user uploads the original, you:

• Return the response as soon as the file is uploaded. The user doesn't need to wait for the server to scale the images.

• Queue the scaling task.

The background loop processes the queue, it reaches the task for the specific image, creates the different versions of the image, and changes the database to indicate that the image is now available in different sizes.

Once the image is processed, its different sizes can be served to the users.

If the image is not processed yet, only the 1:1 variant of the image will be available for the users.

Note that the background task can be distributed among many workers, meaning that it would scale very well. This means that given a decent infrastructure, you can essentially obtain the scaled versions of the image within seconds.