Race conditions in client synchronization

Question

I have a web app whose server creates a Client for each websocket connection. A Client acts as an intermediary between the websocket connection and a single instance of a Hub. The Hub maintains a set of registered clients and broadcasts messages to the clients. This works pretty well but the problem is that a client might miss events between when the server generates the initial state bundle that the client receives on connection and when the client is registered with the hub and starts receiving broadcast events.

My idea is to register the client with the hub before any information is fetched from the db. That would ensure that the client doesn't miss any broadcasts, though now it could receive messages that are already applied to the initial state it receives. To allow the client to disregard these messages I could include a monotonic timestamp in both the initial state bundle as well as broadcast events.

Can you think of a more elegant/simpler solution?

Answer 1

You have a distributed system with multiple concurrent data flows. Those are complicated, but not unsolvable. Due to this distributed nature it is not feasible to assert some before/after relationship between the data sources (DB and message queue). While exactly-once delivery might not be possible, at-least-once-delivery usually is.

Your idea to use a monotonic timestamp is very good, except that a globally monotonic time source might not exist. You can generalize these timestamps to the concept of a message ID, which might already be required by your message queue under the hood (e.g. if it uses the AMQP protocol). A message ID must be unique but not necessarily monotonic, so you can e.g. use UUIDs to generate them.

Then, assuming an ACID database, that messages are written to the database before they are made available on the message queue, and that messages on the queue are delivered in-order:

• The client connects to the message queue. Depending on the queue the queue will now save any pending messages, or we have to consume any arriving messages and store them in a buffer.

• Meanwhile, load the current state from the DB. This state should include the last few messages. We can process these messages and build a set of their IDs.

• While consuming messages from the message queue, discard the message if its ID was already seen in the state from the database.

• Once the first fresh message has been seen, the info about the state can be discarded: all future messages will be fresh.

In this design, the database is the source of truth and the message queue is just a performance optimization for handling multiple consumers. It is worth carefully reading the documentation of your message queue, as it may offer features such as persistence by itself.

Depending on the problem domain you might get additional flexibility. E.g. if you have a chat application like Slack, the client doesn't need to receive the messages in order as long as they are displayed in order (here, a server-provided time stamp is sufficient for ordering). And the server doesn't have to push the chat history through the websocket connection, as the client can request the necessary history itself.