3

I'm aiming to create a simple event-driven system where each microservice operates with its own database. The idea is to share database changes across microservices through events. To ensure proper event ordering, I'm considering a strategy where I use the unique ID of a database row as the event key. This way, events associated with a specific row will be directed to the same partition in a topic. Some message brokers like Kafka maintains the order of events within a partition, this approach seems suitable for preserving event order.

Simple event driven architecture

In a scenario where Microservice 1 has been scaled up to two instances to handle the increased load, and there are two producers in the system, how does a message broker handle the situation where the second instance of Microservice 1 generates a database change and creates an event, quickly followed by the first instance also making a database change on the same row and creating another event? I have a particular concern regarding the potential scenario in which the second producer sends the first event significantly later (due to factors like network problems or waiting for batching), causing it to arrive after the second event has already been dispatched to a partition within a topic.

  • Does the message broker enqueue events as they arrive, even if they are out of order?
  • If yes, are there industry-standard practices to solve problems like this (out-of-order events due to different race conditions), ensuring that events are correctly ordered and processed?

I'd like to understand the best practices for addressing potential out-of-order event issues in a microservices architecture.

Under increased load

4
  • 1
    Questions about specific tools are off-topic. Perhaps you can rewrite it and remove the Kafka related parts.
    – Rik D
    Aug 19, 2023 at 16:07
  • 2
    @RikD, on the contrary, Kafka provides some context, and generally you can replace "Kafka" with "message queue" and the question still stands. I don't see this as a question about coding. It feels like a solid software design problem — how do you handle events that arrive out of order? Aug 19, 2023 at 16:52
  • The question is edited, I’ve retracted my close vote.
    – Rik D
    Aug 20, 2023 at 6:34
  • One way would be to ensure each MS1 instance only handles its own subset of customers. In other words, prevent this: 'the second instance of Microservice 1 generates a database change […], quickly followed by the first instance also making a database change on the same row'
    – Rik D
    Aug 21, 2023 at 7:57

2 Answers 2

1

Kafka stores events (with the same key, assuming your producers use the default partition assignment strategy) in a partition in the order with which they arrive. Think of it as an append-only log.

If your goal is to share database changes, consider using a Source Connector (JDBC Source Connector or Debezium). That will subscribe to your DB and, as changes occur, publish messages into Kafka. Connectors keep track of which rows they have published messages for, avoiding duplicates or out of order messages.

1

Following isn't about how Kafka handles data consistency in a concurrent environment, 'cause that I don't know, it might be it does the same though I don't know. What I do know is to preserve data consistency in a concurrent environment and that is implementing a Turing machine or state machine. Implement a diagram of states with supported state transitions and modify information when the transition is supported. To get fancy and support random order events implement the verification whether the event is a past or future event. When the event is a future event store it until it can be processed. When the event is a past event dump it. After processing an event check whether there are pending events in the future buffer.

Another approach could be to have events with timestamp and process just the events with the timestamp later than timestamp of the information to be modified.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.