The goal is to provide a scaleable system implementing multiple data processing tasks which can be seen as a graph. Data objects will travel that graph. Most object will visit the same nodes in the same order, but depending on the data object properties, the order as well as the included set of processing taks can vary.

Let's assume that every mention of node means a dedicated physical machine and a client is a small piece of code running on a node responsible for retrieving, distributing to the processing tasks, and sending back data.

I found two design principles:

  1. A single task (a single type to be precise, for maximum performance we want as much instances of this task as possible). This gives us some nice optimization potential. Multiple threads of the same task can share common data structures. This is not just a theoretical possibility. Some of our processing algorithms build shared data structures multiple GB in size. The disadvantage I see with this approach is that (for the purest interpretation of this design) we need a dedicated physical machine for every single task we include in our processing chain. In practice, we can just run multiple clients on a single machine and let each client manage its own task. In larger deployments, where multiple nodes are used, this design approach stresses the network a lot, because a single data object needs to visit every node at least once.

  2. Instead of having a single task per client, every client implements all tasks. Advantage: When a data object is received by a client, it can be processed completely without having to travel to other nodes because all processing tasks are done on the same node. Also I assume that this solution scales a bit better, because we don't have to account for different resource consumption of the individual tasks. The full chain of tasks on a single node always needs the same sum of computation power and time, whereas design 1) can maybe run 20 threads of a lightweight task one one node but only 10 threads of a heavyweight task on another node. The disadvantage here is that there is no resource sharing between multiple instances of the same task, so overall resource consumption will be higher.

  3. I said there were only two design approaches, but as always in computer science, there is a middle way. We can group a small number of tasks together and run them on the same node, having multiple instances with shared resources of each task but not as many as design 1 allows.

The metric we want to optimize the design for is throughput, number of data object processed in a given time-frame. Latency is absolutely unimportant. We have the resources for a fair number of nodes and high-speed networks, however, minimizing resource consumption (and a resource here is everything: number of nodes, speed of the network, memory and CPU of the nodes, ...) is our secondary goal. Best case scenario: We can design the system in such a way that we can fine tune between maximum performance and resource consumption for each individual deployment)

I should probably add some more information about how tasks are actually implemented: Long-term goal is to provide tasks as shared libraries implementing a common interface. However as of today, it's much uglier. There are still a lot of tasks implemented in all kinds of ways (Java applications, Python/Shell scripts, tasks that need to be run on dedicated hardware and are accessed via REST, ...). This makes resource sharing much more complicated and sometimes impossible so maybe that should be weighted differently.

I'd love to hear your feedback about these design approaches and what could be an optimal solution. Or maybe you have some ideas what I missed and what factors I should consider additionally when planning this distributed system.

1 Answer 1


I don't think you've correctly identified option 2's downside. Having each node be capable of doing everything doesn't mean you can't do resource sharing. It means every node has to be capable of every task.

That right there can be a hindrance. You don't create specialty nodes that focus on one job. You create nodes that do everything. Done that way scaling up your nodes means scaling up every resource a node requires. If one task requires a license now every node counts against that license. If one task requires tons of system memory now every node requires tons of system memory.

Of course that isn't an argument that every node should only do one task. It's an argument that you should group your tasks on a node wisely.

  • Option 2 is the purest form, having only a single instance per task. Not just being capable of doing everything, but actually doing everything. You're right, of course we can share resources, too, in such a scenario, but I'd say what you described is option 3.
    – flowit
    Dec 9, 2018 at 10:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.