Let's suppose I have a processing workflow of:
Where:
- The workflow can have many instances running at the same time.
- A1 -> B1, C1 -> D1
- A2 -> B2, C2 -> D2
- and so on...
- Each node is an independent process
- Result produced by each node can be stored in a buffer database
- The node
D
relies on the result of bothB
andC
.D1
would depend on the results ofC1
andB1
D2
would depend on the results ofC2
andB2
C
andB
nodes can take different amount of time to finish
My question:
- Is there a name for such system? (Something more specific than just "Distributed system")
- What is the best way to synchronise results of
C1
andB1
in such system- My first idea was to have a single process, which can look at the buffer database and keep checking for both
C1
andB1
to be completed, but as I understand, it becomes a bottleneck as we scale up in number of instances running.
- My first idea was to have a single process, which can look at the buffer database and keep checking for both
All of the research I've done so far talks about components within such distributed system, but I've not found anything to do with joining the data