We currently have an application that is nominally written in Java/Spring boot, but all of the control flow is in NiFi.
For example, there are the following layers in the java application:
- Controllers take an object and return a service call (literally 2 lines)
- Services pass data to a transformer
- Transformers pass data to a converter
- Converters call some API usually (external) and then convert the return object in a response object that goes all the way up the stack to the caller
In the NiFi side, there are a number of process groups that poll various databases, and then feed a huge network of other processors (many are custom .nars). All of the control flow is here (i.e. if an object like this has already be processed, do one thing, else do something else) and while its not very complicated, there is a lot if (all control flow lives here or in the database).
We want to move away from this model of development because it makes it extremely difficult to tell a user when something is actually done (as opposed to what we do now, which is return a 200 when it hits NiFi) and exceptions are very difficult to propagate.
How can we approach this? For scale, we probably have hundreds if not thousands of process groups in NiFi.