I am creating this project based on event sourcing and cqrs. My write and read model are on different databases (and machines) and are connected through an event bus (in particular I am using MassTransit with rabbitmq for the transport)
When an event is generated it is stored in my write model and published on the queue where the read model picks it up and updates the db accordingly
What I still have not completely clear is how to keep the two models in sync in case of errors or problems
For example it could happen that the database on the read side goes down, or messages queue goes down itself and the write side cannot publish any more events
My first approach is, since every aggregate (I am using DDD) has a version number I can use this to keep track of how many events per aggregate instance I have processed.
For example, if in the read side my object with ID 8734 has a version number of 4, and a new event arrives for that object but the metadata says that now we are at version 6, the read notices it missed an event and it can use the message queue to ask for the missing one(s) (This method actually solves also the problem of processing events in order)
The drawback is that the system does not realize that an event has NOT been processed until the next one arrives. (If the event 6 for object 8734 is not generated, I'll never notice that I am missing the event 5)
A second option is: I can create a background processor that 'polls' the database and asks for new events, but this defeats the idea of pub/sub messaging and also forces the read model to process events sequentially, even if they are about different aggreagates where parallel processing would be permitted (and delaying the eventual consistency)
Lastly I could make sure from the read side I keep publishing an event until I receive the ack that the read model (or all the read models) have processed it correctly, but this shifts the repsonsibility on the write side of the application and I do not know if it a good strategy or it is going to bite me back later
How to deal with these situations?