Should TensorFlow prediction be decoupled from post-processing?

Question

I have an API which is basically comprised of two parts: 1. A TensorFlow neural net that provides predictions based on input image (mainly GPU computations) and 2. Post processing on those predictions (mainly CPU)

This is kind of a best practices/recommendation question. What I am wondering is if these two sections of the application should be decoupled, placed in separate Docker containers and scaled separately. There is no other use for the TensorFlow predictions (no other apps would want to receive predictions directly so there is no need for decoupling in terms of accessibility).

The only scenario I can think of that would warrant decoupling is if the Post-Processing consumed a large amount of CPU resources that forced the application to scale when the GPU was being underutilized (the prediction part of the app was handling the load just fine) and by forcing the application to scale we are using more GPU resources than necessary.

However as long as sufficient CPU resources can be allocated to the server so that the point at which the app scales is a point of high utilization on both the CPU and GPU I would see no reason why the services should be decoupled.

Hopefully this makes sense - any suggestions?

Answer 1

From experience at my current job, decouple as soon as you can. Design things as far apart as possible. We have some similar workflows at my current job from the sounds of it so let me give an anecdote from my world.

In the begining we did A and B, B was dependent on A so it made sense to just throw them together. Then we started doing C which was really independent of A and B, but hey, it's easier to just throw it right after B than to rework some small aspects of our product. Then one day D and E showed up at the door. D was independent of everything at this point but E requires C and D. The effort to now split up the work is much greater than when we had just A,B and C. So we just throw D and E in line.

Oh look F,G,H, and J just walked through the door, each with a big pile of money; now we need to work fast, no time to rework things. Who cares three of the 4 are independent and by throwing them into the mix one after one another we are doing huge amounts of unneeded work.

Now we do all the work for everybody, but clients only pay us for what they want to see so we just don't return the parts of the response a client does not pay us for.

(BTW, sounds like I am ragging on my employer, but I love the work. Thankfully I work almost exclusively on the individual modules and not on the glue sticking them together, the people working mostly on the glue though do have a bit of a different attitude which is a different story altogether.)

Answer 2

My preference will be decoupling, since you will not need post processing for every image you have. It will give you a more flexible portfolio of services for future optimization and scalability as you stated.

Your desired functionality, requirements for speed and resource allocation will decide it though. You can start with one implementation and give a month for a dry run, and if you are not happy with your results you can revert to the other too. Best option might be giving a unified interface to the user but implementing in a decoupled fashion.