The goal is to provide a scaleable system implementing multiple data processing tasks which can be seen as a graph. Data objects will travel that graph. Most object will visit the same nodes in the same order, but depending on the data object properties, the order as well as the included set of processing taks can vary.
Let's assume that every mention of
node means a dedicated physical machine and a
client is a small piece of code running on a node responsible for retrieving, distributing to the processing tasks, and sending back data.
I found two design principles:
1) a single taks (a single type to be precice, for maximum performance we want as much instances of this task as possible). This gives us some nice optimization potential. Multiple threads of the same task can share common data structures. This is not just a theoretical possibility. Some of our processing algorithms build shared data structures multiple GB in size. The disadvantage I see with this approach is that (for the purest interpretation of this design) we need a dedicated physical machine for every single task we include in our processing chain. In practice, we can just run multiple clients on a single machine and let each client manage its own task. In larger deployments, where multiple nodes are used, this design approach stresses the network a lot, because a single data object needs to visit every node at least once.
2) Instead of having a single task per client, every client implements all tasks. Advantage: When a data object is recieved by a client, it can be processed completely without having to travel to other nodes because all processing tasks are done on the same node. Also I assume that this solution scales a bit better, because we don't have to account for different resource consumptions of the individual tasks. The full chain of tasks on a single node always needs the same sum of computation power and time, whereas design 1) can maybe run 20 threads of a lightweight task one one node but only 10 threads of a heavyweight task on another node. The disadvantage here is that there is no resource sharing between multiple instances of the same task, so overall resource consumption will be higher.
3) I said there were only two design approaches, but as always in computer science, there is a middle way. We can group a small number of tasks together and run them on the same node, having multiple instances with shared resources of each task but not as many as design 1 allows.
The metric we want to optimize the design for is throughput, number of data object processed in a given timeframe. Latency is absolutely unimportant. We have the resources for a fair number of nodes and highspeed networks, however, minimizing resource consumption (and a resource here is everything: number of nodes, speed of the network, memory and CPU of the nodes, ...) is our secondary goal. Best case scenario: We can design the system in such a way that we can fine tune between maximum performance and resource consumption for each individual deployment)
I should probbaly add some more information about how tasks are actually implemented: Longterm goal is to provide tasks as shared libraries implementing a common interface. However as of today, it's much uglier. There are still a lot of tasks implemented in all kinds of ways (Java applications, Python/Shell scripts, tasks that need to be run on dedicated hardware and are accessed via REST, ...). This makes resource sharing much more complicated and sometimes impossible so maybe that should be weighted differently.
I'd love to hear your feedback about these design approaches and what could be an optimal solution. Or maybe you have some ideas what I missed and what factors I should consider additionally when planning this distributed system.