0

Let's suppose I have a processing workflow of:

Where:

  • The workflow can have many instances running at the same time.
    • A1 -> B1, C1 -> D1
    • A2 -> B2, C2 -> D2
    • and so on...
  • Each node is an independent process
  • Result produced by each node can be stored in a buffer database
  • The node D relies on the result of both B and C.
    • D1 would depend on the results of C1 and B1
    • D2 would depend on the results of C2 and B2
  • C and B nodes can take different amount of time to finish

My question:

  • Is there a name for such system? (Something more specific than just "Distributed system")
  • What is the best way to synchronise results of C1 and B1 in such system
    • My first idea was to have a single process, which can look at the buffer database and keep checking for both C1 and B1 to be completed, but as I understand, it becomes a bottleneck as we scale up in number of instances running.

All of the research I've done so far talks about components within such distributed system, but I've not found anything to do with joining the data

2
  • 1
    Regarding the name, “event driven” might be more specific than “distributed”. Such systems commonly assign correlation IDs to events/messages so that different events relating to the same original event can be linked.
    – amon
    Apr 21, 2022 at 15:36
  • That's a good point @amon. Is there any research or articles that you've encountered about approaches of event consolidation in such systems?
    – August
    Apr 22, 2022 at 11:40

1 Answer 1

2

About the name of such a system I can't say anything - I'd guess that it's not a specific name but a function of the system, which I would informally call event consolidation.

Regarding the behavior of node D, that's actually very simple: It processes events from both B and C, and on each event it looks into its local buffer to see whether the matching event from the other source was already seen. If yes, both are present, so D can do its calculation, if not, the event is stored into its local buffer for later lookup. You will need some timeout to handle events that were sitting in the buffer for too long, which could mean that the matching event could not be generated due to some problem.

If you want to scale up, you can do that via sharding - the index of the event determines to which instance of D it goes, and the other event, having the same index, will naturally go to the same instance.

To handle concurrency within D, you need to use one of the possible patterns - easiest would be to have a queue of incoming events that is processed by a single event reader process, but you could also have a mutex-protected buffer when there are multiple event handlers, so each one has exclusive access to the buffer while looking up a matching event.

1
  • Thanks for taking the time to answer Hans. I agree that having a single event reader process is certainly the most straight forward approach. I will go ahead and only introduce mutex-protection if the reader becomes a bottleneck. Thank you for your insights.
    – August
    Apr 21, 2022 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.