I believe there's three typical ways you could handle this:
- Guarantee delivery of your messages
- Run a reconciliation process
- Switch to a pulled events approach
A little detail on each approach...
Guarantee delivery of your messages
This is for when you really want to make sure that your messages get through, and as soon as possible.
Firstly, use the transactional outbox pattern in the
user service to ensure that all messages that should be sent as a result of a database transaction are successfully sent to the message broker.
Secondly, design and deploy your message broker to have high availability and (more importantly) very high durability (i.e. 99.99....?% of messages are not lost).
Thirdly, ensure that messages are not ACK'd by the
auth service until they've been processed and the results committed to your db (the receive-side analog of the transactional outbox).
If it's really, really important that you never lose messages, you might also want to keep a Sent Messages log file at the
user service. In the case of data loss in your message broker, you can then replay messages from the log.
Run a reconciliation process
If you're okay with the "eventual" of your eventual consistency being a little longer, and the amount of data that's shared between the services is not prohibitively large, you can run a reconciliation process. This would typically be done either by the
user service regularly exporting a dump, or by the
auth service regularly requesting all the data owned by the
user service which it is caching. Either way,
auth regularly receives
user service's full picture of the world and can either update itself if that's relatively easy to do or alert humans to intervene if it detects an inconsistency that can't be automatically handled. It's a good idea to have a method of ensuring that you're not overwriting changes in
auth from recently received messages with stale data from
user that is from before the message was sent.
Switch to a pulled events approach
This one's an alternative to using a message broker, so a little orthogonal to your question, but it's worth considering. Instead of pushing events through a message broker, you can provide an
events/ endpoint on your
user service. The
auth service then becomes responsible for knowing where it is up to in the events stream, and for processing messages in order and calling developers for help if it can't understand the data it receives.
Avoiding the problem in the first place
You do want to design for the day when this kind of error case to occur. But it's a good idea to also put into place practices that will greatly lower the likelihood. One such practice you probably want to look into is consumer-driven contracts, which essentially try to catch such errors at build time by breaking the build.
Handling the problem well
Also, when the problem does happen, it's nice to handle it gracefully. This is typically done using what's called a Dead Letter Queue, where any message that causes an error at the receiver gets removed from the inbox and placed in a separate queue. The messages are then kept in that queue until something is changed in the system to allow them to be processed again. In your scenario, you would probably push all the DLQ messages back into the inbox after deploying a new version of the
auth service that has been updated to understand the new message format.
 I say "caching" here because the data should only ever be owned by one service. It seems
user service owns this data, and
auth service is caching it for its own purposes. While this duplicates data, it's a good pattern because it increases service autonomy.