1

Given a Set of messages sent to a Queue, what are good ways to determine all of the messages have been processed?

Constraints:

  • Large number of messages
  • Other messages besides those in the Set will be in the queue (queue size may never be 0)
  • Queue does not guarantee ordering of messages

The best solution I have thought of so far is: give each message a GUID, when posting message to the Queue also add an entry to an "Unprocessed" table with the GUID key and a "SetId" column. When the message is processed, delete the "Unprocessed" table entry for that GUID. Query if all messages have been processed by "Any(unprocessed where SetId={id})".

The reason I am asking is for the following workflow: the user triggers an automation, which itself triggers numerous events for actions to perform on the system, and then we want to display to the user whether the automation has completed. Backend microservice system. Open to alternative techniques besides the general problem I proposed as well. I am also open to suggestions like: choose a different queue that enforces ordering, if that is a practical solution you would use.

1

Some message processing frameworks (e.g. Celery) already include tools for background task management that allow task joining, chaining and whatnot. Otherwise, it should not be too hard to implement or extend (e.g. ASP).

Your solution seems adequate for given use case, implementing something more advanced would be an overkill. But if you ever find yourself looking for more and having to roll your own implementation, here are few suggestions:

  • Task chain is a graph structure. Every tasks depends on completion of previous tasks, if any.
    • You may connect them with a GUID or other kind of a cookie (this.NextLinkId == this.Next.PreviousLinkId, vice versa) to avoid setting up a many-to-many relationship table.
  • For tracking and debugging purposes, only delete entire chain and only when it's fully complete.
  • To enforce ordering, just make a flat chain, otherwise do a join. Actually, do whatever makes more sense, there are no limits now.
2
  • Thank you! This is very interesting point about the graph structure if more sub-tasks were used, and the practical consideration to wait for deleting the chain until fully complete. What does M2M mean? – MattHH Apr 2 at 15:56
  • 1
    @MattHH You are welcome! M2M is a shortcut for "many to many". Regarding graphs, I see background task management identical to normal async/task/future/promise chaining, if implemented well. – Shadows In Rain Apr 2 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.