What do you do if the consumer that was handling an order crashed?
You do not want to have a consumer receive items on an order and process them. You want an intermediary that builds partial orders with the items that are being pushed.
So, on one side you have a producer that begins an order, add items to the order and finishes the order. In the middle you have the intermediary that aggregates the items into partial order objects. In the other side you have the consumer of orders.
Deciding what design to implement:
- Pull vs Push
- Synchronous vs. Asynchronous.
Pull vs Push
Push based design requires a simpler threading model (I go back to this), it is harder to get it wrong.
The Push based design will just be doing a method call to notify consumers. Which means it will be aware if they fail (e.g. thrown exception). And this makes it easy to have it push the value to a different consumer if a consumer crashed.
If we do that at the level of items being pushed into the order, then it will be very hard to go to a different consumer. It would require to keep track of partially completed orders. The intermediary solves this problem. Once we have the intermediary it makes much more sense to aggregate an order and process it as a single value.
On the other hand, if the consumer fails on a Pull based design, there is no easy way for the producer to know. Which means that having the producer give the same value to a different consumer is hardly viable.
Distributing the Pull based design would result in clients pooling the server, which is not good. However, the Push based design is much easier to distribute... for example, the method would do an asynchronous call across the network (which could be sockets, RPC, web design, message queues, etc...).
Finally, if the consumer needs to pull, it is easy to have the Push based design add to a queue, per consumer, from where it can pull (although that loses some of the benefits). On the other hand, if the consumer wants notifications being pushed, with the Pull based design, now you need to have a thread pulling and pushing, per consumer, which is more expensive.
I strongly believe a Push based design is the superior option. However, if I understand correctly, you are currently using a database. The way databases work is that you Pull, I mean, query. Thus, I assume you have a Pull based design. Changing to a Push based design will have a cost in refactoring your code.
Synchronous vs. Asynchronous
The intermediary step must be Synchronous. Having it asynchronous is just making it harder to get right and maintain.
The producer side can be Asynchronous. However, this does not have to exposed in the design, since it will likely be fire and forget.
The consumer side, since I am encouraging a Push based design, it will be Synchronous.
Therefore, as far as the API goes, it will be Synchronous.
Implementation considerations
You will have multiple queues, either way. If we assume no intermediaries, you need: One queue of items per order, one queue of orders that are yet to be assigned to a consumer (If you are pulling, you pick the order from here. If you are pushing, new subscribers can take from here), and – depending on the design – a queue or set of available consumers. The intermediaries would replace the queues of items per order.
Note: Since the Push based design needs to find a subscription to notify, it might use a set of consumers that are available.
The Pull based design will need wait handles to have the consumer threads wait until new values are made available (this is why I say the Push based design has a simpler threading model). Also consider tear-down. We need to make sure those thread are not stuck waiting.
On the Push based design, consumers are not tied to a thread. Instead, we use subscriptions to keep track of them. A subscription must be a reference to (or an object with an interface that has a) method to call to pass a value.
It is a good idea to break these solution into components. In particular, the Pull based design, effectively has blocking queues (queue plus wait handle), and the Push based design, effectively has one way channels (subscription plus queue). Those are things you might get from libraries, and using them will simply the implementation.
Note: on the Push based design, if there are no subscription to push to, the values got to be queued.
You still need to be extra careful about race conditions. In particular, the Push based design has two notable race conditions:
- A new subscription when there are no unassigned happening while a value is being pushed could result in the value being queued and the subscription not being assigned.
- (in some designs) A new subscription, finished going over the queued values and removes the queue, while a new value is being pushed into the queue could result in the value being lost.
There are a few useful locking solutions for these. In particular, I want to mention that you can queue operations. That is, add the operation (it can be a callback) to do to a thread safe queue. Try to get the lock. The thread got the lock executes all the operations in the queue. If the thread did not get the lock, it mans another thread got it... and that other thread will do the operation. Thus it can move on without waiting. This way no thread will be waiting on the lock. Use only on identified race condition. If you do this for everything, it would be very easy to have a thread stuck inside the loop indeterminately.
Note: the Pull based design gets around its race conditions by retying. It is try-wait-loop.
