1

Consider a social network in which a user photo is needed in different resolutions large/small/medium etc.

When the photo is uploaded, I do not want to make the user to wait too long for the image to be processed and the different urls for each size to be returned.

I want just to return the base url ex domain.com/images/something.jpg Then when the frontend calls for domain.com/images/something.jpg?s=large I generate the large image, cache it somwhere and always return that.

The problem that I cannot figure out with the above approach is that I cannot use s3 anymore without double transferring the data.

With the above aproach first I need to transfer the data from s3 -> backend server then from backend server -> frontend.

If I go withe the solution that I do not like, frontend waits longer for backend to generate all image sizes and return all urls, then only one transfer need to be done, from s3 to frontend.

I also thought of another possible solution that backend returns urls withhout the image being avaible, for example:

[
  large: domain.com/dd.jpg?s=large
  small: domain.com/dd.jpg?s=small
]

Then when the frontend calls for example the small url, if it does not exist first is generated, then stored in s3 then backend returns 302 redirect to the s3 url. I think that this approach will be bad because of many 302 redirects, but offers the flexibility that I want.

I am still trying to figure out how to solve the problem. Any suggestions please ?

1

If you don't want the users to wait on upload, you probably don't want them to wait when requesting the specific size of an image either (since in general, the user who uploaded the image would be the first one who will request it anyway). In this case, there is a third solution: generate the scaled versions of an image through a cron job or a background task.

When the user uploads the original, you:

  • Return the response as soon as the file is uploaded. The user doesn't need to wait for the server to scale the images.

  • Queue the scaling task.

The background loop processes the queue, it reaches the task for the specific image, creates the different versions of the image, and changes the database to indicate that the image is now available in different sizes.

Once the image is processed, its different sizes can be served to the users.

If the image is not processed yet, only the 1:1 variant of the image will be available for the users.

Note that the background task can be distributed among many workers, meaning that it would scale very well. This means that given a decent infrastructure, you can essentially obtain the scaled versions of the image within seconds.

  • This is essentially how YouTube reprocesses uploaded video on it's network. Not sure of the OP's environment, but I worked on a project where we used Spring Cloud Data Flow for the background image processing, and had the flow start with the new file being uploaded, and ending with a service that associated the variants with the origin object. – Berin Loritsch May 5 '18 at 20:57
  • @Arseni Mourzenko thanks for the reply. What about the case in which the frontend asks immediately for an url and the background worker has not yet finished creating that size of image ? – Kristi Jorgji May 5 '18 at 21:45
  • @KristiJorgji: "If the image is not processed yet, only the 1:1 variant of the image will be available for the users." – Arseni Mourzenko May 5 '18 at 21:47
  • How can I do that mapping by using s3 and without having to double transfer the data. That is my main concern, I do not want domain.com/image.jpg?size=size do some login then fetch image from s3, then give this image back to frontend. The data will be transfered twice (image) and we pay for the data. Also I need to return 3 urls for all sizes to the frontend, and if the frontend calls for a size and that size is not yet available it will result in 404 (in case of s3 I do not have any url to return before uploading the processed image there) – Kristi Jorgji May 5 '18 at 21:49
  • When the HTML page containing an image is generated, the web application knows (from the database) whether the scaled versions of the image are ready or not. Depending on that, it puts either image.jpg?size=... or image.jpg?original in the <img/> tag's src. – Arseni Mourzenko May 5 '18 at 22:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.