Websocket client reconnection best practices

Question

I wasn't sure whether to post this on StackOverflow or here. Let me know if you think it belongs on StackOverflow.

I have an application that will run semi-persistently, i.e. all the time. The application interacts with a financial service through a websocket connection (we are the client-side). We keep the connection alive to remove connection latency. If, for whatever reason, the websocket disconnects, we should try to reconnect.

Logically, how should we go about attempting reconnection? This is such a common function, there must be some best practices and techniques. I can find lots of examples of how to implement the trivial strategy of just trying to reconnect over and over, with a small interval in between. I have found at least one tool that uses a slightly more complex strategy. I have not found any discussions of the topic in the abstract.

Are there best practices or some theory about the best way to reconnect websockets in general? I feel the obvious approach probably works, but smart people can often make things way better than the obvious solution. In particular...

• Should we attempt to reconnect forever or should we give up and shut down eventually? If we should give up eventually, when?
• Should we vary (i.e. increase) the interval between reconnection attempts?
• When the app tries to use the websocket while it is not connected, should we backlog that interaction and send on reconnect or should we throw an error?
• Should our reconnection attempts be independent of attempts to use the websocket, or should we try harder to reconnect if the app is trying to use the websocket?

I feel there must be some theoretical treatment of these questions. I imagine the answers to these questions would not be "yes" or "no" but rather a discussion of how to decide.

Answer 1

Network connections are not reliable, so it's important to consider how you will handle interruptions and gracefully recover.

Backoff strategy

For reconnecting, exponential backoff is typically an optimal strategy. This is because what happens when your servers are down for some reason and are bombarded with requests that will all fail. You would like to not DDOS your servers with your own application, so you want to limit the rate of incoming requests. If the backoff interval is at least doubled after each attempt, then the requests per second hitting your server will eventually reach a steady rate and not grow uncontrollably, regardless of how many clients there are. It could however be considered to limit the maximum interval e.g. to a couple of hours.

When using exponential backoff, it's also unnecessary to stop trying at some point. The intervals between retries just grow longer.

From an UX perspective, it would still make sense to offer a button to manually retry. This is especially valuable if the cause of the interruption is client-side and not server-side, for example if the user's internet connection was manually restored.

Data synchronization strategy

The other aspect from recovering from interruptions is how to re-sync with missed messages.

In a way, this is equivalent to the problem of loading the app the first time, or refreshing the web page with F5.

From a reading client perspective, it probably makes sense to do the following when a connection is re-acquired:

• Start subscribing to events via WebSockets. Store all real-time messages in a queue.
• Make a batch request to fetch the current state. Load the current state into the application and enable interactivity for the user.
• Process the enqueued messages as if they had just arrived. If a message is already part of the loaded state, discard it. This requires that all messages have an UUID or other unique ID, or can otherwise be applied in an idempotent manner.
• Switch to processing new messages in real-time as they come in.

From a writing client perspective, it would also be necessary to send user-generated events that were created while the application was offline. Depending on the purpose of the app, it could make sense to prevent the creation of further events while the client is offline. Show an error, lock up the UI. Alternatively, an offline-first design could make sense. A web application would then update locally stored application state while offline, and synchronize the local state with the server once the connection is re-acquired.

However, synchronizing databases can be really difficult. A theoretical approach is to use CRDTs, but it's not always applicable in practice. Some problems can be represented via time-stamped event streams which can be later merged. Other problem domains would benefit from presenting an interface to the user where the user can resolve conflicts, similar to how Git works. In practice, a common conflict resolution algorithm is “latest change wins” but this can lead to data loss. There really is no best solution, and this is highly context-dependent.