What are the approaches for joining data in distributed processing

Question

Let's suppose I have a processing workflow of:

Where:

• The workflow can have many instances running at the same time.
  • A1 -> B1, C1 -> D1
  • A2 -> B2, C2 -> D2
  • and so on...
• Each node is an independent process
• Result produced by each node can be stored in a buffer database
• The node D relies on the result of both B and C.
  • D1 would depend on the results of C1 and B1
  • D2 would depend on the results of C2 and B2
• C and B nodes can take different amount of time to finish

My question:

• Is there a name for such system? (Something more specific than just "Distributed system")
• What is the best way to synchronise results of C1 and B1 in such system
  • My first idea was to have a single process, which can look at the buffer database and keep checking for both C1 and B1 to be completed, but as I understand, it becomes a bottleneck as we scale up in number of instances running.

All of the research I've done so far talks about components within such distributed system, but I've not found anything to do with joining the data

Answer 1

About the name of such a system I can't say anything - I'd guess that it's not a specific name but a function of the system, which I would informally call event consolidation.

Regarding the behavior of node D, that's actually very simple: It processes events from both B and C, and on each event it looks into its local buffer to see whether the matching event from the other source was already seen. If yes, both are present, so D can do its calculation, if not, the event is stored into its local buffer for later lookup. You will need some timeout to handle events that were sitting in the buffer for too long, which could mean that the matching event could not be generated due to some problem.

If you want to scale up, you can do that via sharding - the index of the event determines to which instance of D it goes, and the other event, having the same index, will naturally go to the same instance.

To handle concurrency within D, you need to use one of the possible patterns - easiest would be to have a queue of incoming events that is processed by a single event reader process, but you could also have a mutex-protected buffer when there are multiple event handlers, so each one has exclusive access to the buffer while looking up a matching event.