I have a system where a set of Operations needs to be done every day. Each Operation has the following metadata: Type, Entity and a Date (independent of current date). It reads something from the database, computes stuff and saves results to the database. An Operation is a combination of: [Entity, Type, Date]. I need the highest degree or parallelism (or distribution) possible.
Operations with different Entities can be done independently (order does not matter). The Type however (lets' say there are Types T1, T2, T3) need to be processed in order. The Date (within a Type) can be done independently (again order does not matter). All Operations for a single Entity can be created (are known) at the same time (generated once per day, multiple entities at different times of a day). The amount of Types and Entities is a known constant.
So let's say there is a set of operations:
#1 [E1, T1, D1], #2 [E1, T1, D2], (different dates) #3 [E1, T2, D1], #4 [E1, T2, D1], (different type than the above) #5 [E2, T1, D1] (different entity than those above)
I can start #1, #2 and #5 in the same time. Then I need to wait for #1 and #2 to finish (they both have T1 which is < than T2). Only then can I start (again in parallel) #3 and #4.
By parallel or distributed I mean distribution over many threads within one machine as well as distribution over multiple machines. There can be many Operations with the same Entity and Type (different dates), but also many Types and so on.
What are the possible (general) architectures allowing me to do the above?
What I have now:
Everything is put to an Azure Service Bus (now it's a single queue). An Operation is a single message. There are many services consuming the messages and performing actions. I do not know how to synchronize the messages with different Types. Azure Service Bus (afaik) does not let me pick some kind of messages before others. It just gives me the 'next available' message.
Using separate queues for different Operation Types. When my processors want a message, they look at T1 queue (and take if anything is there), if it's empty they look at T2 queue (again take if anything is there) and so on. This way I have forced the Type constraint. Problem #1 is that I do not know when the last Operations have finished (the queue being empty means the work has started, not finished). Problem #2 is that I have to wait for all operations of type T1 to finish (which for example are more database-intensive) and all T2 (potentially CPU-intensive) are done at the same time although they could be parallelized for different Entities.