0

How would you go about making sure that "events" sent as messages to any sort of broker (Kafka, RabbitMQ etc) from multiple instances of the same service (load balanced) are actually added to the "queue(s)" in a correct order?

So for instance.. Say we have a created an application/service. The service publish the following events/topics as a message to a broker:

  • UserCreated
  • UserUpdated
  • UserRemoved

Now since we have a heavy load on the service, we decide to scale the service to multiple instances.

Some action is taken that ends up creating a UserCreated event, and directly after another action is taken which ends up creating a UserUpdated event.. however both the events are "created" by different instances of our service. Now lets pretend that the service creating the "UserCreated" event is for some reason running slower, so the UserUpdated event is actually added to the queue before the UserCreated event.. is there anyway to prevent this? Any patterns or something that could read up on to understand how to make sure that the order of messages/events actually ends up correctly in the queue in a that has multiple publishers?

Please notice that the above example is just that.. an example, Im fully aware that we could probably "design" our messages in the example above so that UserUpdated sends the fully updated user, and if it doesnt exist we create it.. and if the CreatedUser with the same ID/Username occurs after the UpdatedUser we could just discard that event, but thats not the problem I need to solve. Im just wondering of how to solve the issue of making sure that events are handled by a consumer in the correct order (which I assume required the messages to be published in a correct order?).

Improve this question
16
  • It sounds like you're trying to use a "fire-and-forget" message pattern in a situation which isn't suitable for fire-and-forget -- i.e. you need strong guarantees of exactly-once delivery and delivery ordering; fire-and-forget is simply the wrong pattern to use in this case. Ideally choose a different pattern instead (different broker technologies may have alternatives built-in). For example, RabbitMQ has a feature for Publisher Confirms: rabbitmq.com/confirms.html#publisher-confirms
    – Ben Cottrell
    Oct 24, 2021 at 10:42
  • You could research if the Actor pattern is a good fit for your system. It would ensure all actions on a user (or whatever your actual entity is) are handled by the same actor.
    – Rik D
    Oct 24, 2021 at 11:04
  • 1
    The Theory of Relativity tells you this can't be done. Seriously. A way to think about this is to consider what really happens in a human workflow in the same scenario. Typically either a single human is designated to handle all UserPortfolioModification events - and that human uses his judgment to recognize the situation and deal with it when >1 modifications land in his inbox at the same time - or even simply the same day - or there are >1 humans processing, each can recognize the situation, and they kick it upstairs to someone with judgment. You've got to model this somehow.
    – davidbak
    Oct 24, 2021 at 17:17
  • (The only differences between a computer workflow and the same workflow when human powered are 1) speed at which everything happens and is expected to happen, and 2) that in the human workflow there are humans in the loop to handle exceptions, even when the exceptions aren't acknowledged to be possible in the workflow! There are always these unacknowledge exceptions that humans just fix with their knowledge and judgment and that's where you run into trouble with computer-only workflows, that don't recognize that shit happens and needs to be fixed. (You're acknowledging it!)
    – davidbak
    Oct 24, 2021 at 17:22
  • 1
    I don't think you can publish messages in "the correct order" from separate machines. I don't think you can expect the queue mechanism to resolve it: it doesn't have the brains or the context. I think you need an intermediary service that fixes this problem (out of order messages) according to the semantics you desire and accept that it will introduce latency too. But I don't have any references for you on typical ways (much less easy ways) this is accomplished, hence: no answer from me, just comments.
    – davidbak
    Oct 24, 2021 at 18:47

2 Answers 2

Reset to default
1

I would not, instead I would ensure they are processed in the correct order.

This is much easier, you just listen to the queue and put out of order messages aside until you find the missing earlier message then process them in order.

Do this with an intermediary "routing" message processor. Which picks up off the incoming queue and routes to "process now" or "messages on subject X that need to wait for an earlier message" queues.

Unfortunately this router can't be stateless as you need to keep track of the message groups to some extent. So it can become a bottleneck if you need to order many messages over long periods of time.

Improve this answer
2
  • Also introduces latency, unavoidably (else how could you even determine the second message to arrive should have been first without holding the first to arrive).
    – davidbak
    Oct 24, 2021 at 19:34
  • I guess you could argue whether that latency was "introduced" by the solution, or by the requirement to process in order
    – Ewan
    Oct 26, 2021 at 12:17
0

Assuming that these service instances share the same database, you can use the outbox pattern in combination with a transaction isolation level that protects you from all your possible race conditions. If you place the "UserCreated" event persistence on your outbox within the same transaction than the user creation itself, you can warrant the correct order of the messages written to the outbox, regardless of which service instance added the message.

Please keep in mind that this doesn't deal with the hypothetical problem of the message consumer receiving the messages in different order, only in-order message delivery to the broker.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.