I have an API which is basically comprised of two parts: 1. A TensorFlow neural net that provides predictions based on input image (mainly GPU computations) and 2. Post processing on those predictions (mainly CPU)
This is kind of a best practices/recommendation question. What I am wondering is if these two sections of the application should be decoupled, placed in separate Docker containers and scaled separately. There is no other use for the TensorFlow predictions (no other apps would want to receive predictions directly so there is no need for decoupling in terms of accessibility).
The only scenario I can think of that would warrant decoupling is if the Post-Processing consumed a large amount of CPU resources that forced the application to scale when the GPU was being underutilized (the prediction part of the app was handling the load just fine) and by forcing the application to scale we are using more GPU resources than necessary.
However as long as sufficient CPU resources can be allocated to the server so that the point at which the app scales is a point of high utilization on both the CPU and GPU I would see no reason why the services should be decoupled.
Hopefully this makes sense - any suggestions?