I'm aiming to create a simple event-driven system where each microservice operates with its own database. The idea is to share database changes across microservices through events. To ensure proper event ordering, I'm considering a strategy where I use the unique ID of a database row as the event key. This way, events associated with a specific row will be directed to the same partition in a topic. Some message brokers like Kafka maintains the order of events within a partition, this approach seems suitable for preserving event order.
In a scenario where Microservice 1 has been scaled up to two instances to handle the increased load, and there are two producers in the system, how does a message broker handle the situation where the second instance of Microservice 1 generates a database change and creates an event, quickly followed by the first instance also making a database change on the same row and creating another event? I have a particular concern regarding the potential scenario in which the second producer sends the first event significantly later (due to factors like network problems or waiting for batching), causing it to arrive after the second event has already been dispatched to a partition within a topic.
- Does the message broker enqueue events as they arrive, even if they are out of order?
- If yes, are there industry-standard practices to solve problems like this (out-of-order events due to different race conditions), ensuring that events are correctly ordered and processed?
I'd like to understand the best practices for addressing potential out-of-order event issues in a microservices architecture.