1

In a distributred system that uses parallel processing to complete tasks and I want to determine when all sub tasks have been completed, what design method or principals can i apply ?

I am using RabbitMQ to consume messages that are generated from a single process, each consumer can generate multiple messages to other RabbitMQ queues that can be processed in parallel. I want to be able to determine when all tasks generated from the first task are finished, how can i achieve that ?

2
  • I love this site, i ask a question immediately get voted down and zero comments as to why ? maybe you could ask some questions via comments to help me improve my question or maybe even just explain why ive been down voted !?
    – Manse
    Feb 23, 2023 at 12:18
  • 1
    I’ve been thinking about this question recently and appreciate you asking it!
    – COleson
    Nov 24, 2023 at 21:09

1 Answer 1

4

Do you want to poll (repeatedly ask "are you finished yet?") or wait on a blocking primitive?

The first is simple: Have a database table where each row represents a task. One column is the current status. As long as one of the rows has a "busy" status, you're not done yet.

The second is also not difficult: Create a "finished" queue where each task places an "I'm done" message. The process that should wait for the completion of all tasks just needs to know how many tasks were spawned, and retrieve that many messages from the queue. Once the last message has been retrieved, you know that all tasks have been done.

3
  • Great suggestions, thank you for your answer. Waiting for a blocking primitive is the preferred method in my case so the second option is better
    – Manse
    Feb 23, 2023 at 13:30
  • 1
    Pure waiting's fine so long as receiving the completion message is 100% reliable. If you cannot guarantee receipt, you may have to manage a list of dispatched requests that are updated when the reply is received (Option 1) with a periodic thread to re-send or query anything that hasn't returned in a reasonable period.
    – mcottle
    Feb 27, 2023 at 3:25
  • True, if the underlying system isn't reliable you need to have a problem resolution strategy. If the tasks are idempotent, retrying may be automatic, but otherwise it may require human analysis and decision, and the only automatic thing should be a notification if things take longer than expected. Feb 27, 2023 at 5:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.