This question is not specific to images, but this is the current issue im having so im using this as an example.
So basically, I have some kind of a pipeline that some data(in this case, an image), has to go through before It can be used in my app(in this case, a web app).
If we split the pipeline into functions it would look like this: 1. The image is uploaded to a storage(cloud storage). 2. The image is converted into multiple formats and sizes. 3. All of the different sizes and formats are registered in the database in relation to the original file.
My question is, what would be the right way of managing this process, and keeping track of errors, retries, updating the front-end etc.
I think im trying to solve this task with the wrong mindset, and thus im getting a bad solution. But maybe functions is not the right way. Maybe I need to opt out of cloud functions, and use some other serverless solution, like for example "Google's AppEngine".
Currently, I have a function with an HTTP endpoint that: 1. Receive the file and stream it into the Cloud storage(I do this because I try to avoid a heavy front end). 2. Save the name of the file in the DB with the temporary name, and status(status is currently waiting for optimization/cropping). 3. The upload into Cloud Storage triggers another function that starts the optimization and cropping of the image. 4. Update the DB with the status and all of the subfile names. 5. delete the original file.
The problem is that between 2 and 3, there is a broken link. I can't really know what is going on on the client side without multiple requests to the DB to see the status of the image.
So is this the right way? is there any better way? I found some similar problems all over my app development, like for example with data validation, and that made me think that maybe im getting it wrong.