5

I have been looking at different solutions for large scale chatting solutions. I feel as if I understand 90 % of it but am turning to this forum to tie the knot.

I imagine running a bunch of message servers behind a load balancer, keeping long lived connections to the client (wss or xmpp). The amount of servers can scale horizontally based on incoming requests. Then on the backend the messages are distributed using a pub-sub pattern. Meaning if two clients are trying to communicate but are connected to different servers, then the pub-sub messaging distribution will take care of it.

So far, it all make sense. The image below clarifies it perfectly.

enter image description here

But I can't imagine how each WebSocket Server is managing to subscribe to all of the topics the PubSub server (redis) will handle.

I imagine the PubSub server managing 100,000 requests per second through sharding. But I can't understand how every WebSocket Server then handles the subscriptions. Because being subscribed to all topics on all shards will that not flood each WebSocket Server?

So once we go down that path of sharding, how do we distribute messages properly? Because then we need some intelligence in order for each WebSocket Server to be subscribed to the exact right topics where client A and client B is sending its messages. Somehow each WebSocket Server needs to be continuously subscribing/de-subscribing based on which topic is being used for client A and client B.

Or is actually the WebSocket servers capable of being subscribed to EVERY topic? Am I wrong in seeing this as a problem? If so, then why use load balancing at all?

Edit: Taking inspiration from here and here

Edit2: rephrasing.

10
  • 3
    surely a pub/sub message queue solution will easily scale to 100,000 rps?
    – Ewan
    Mar 18, 2022 at 19:23
  • 2
    the websocket server is limited to the number of users it supports right? so there is a natural limit to the number of messages. I guess with no restrictions, you could have a single user sign up to every other message on the system? but they wouldnt be able to read them all
    – Ewan
    Mar 19, 2022 at 9:51
  • 1
    what's the throughput on your load balancer?
    – Ewan
    Mar 19, 2022 at 9:59
  • 2
    If the number of clients per WebSocket server is very small compared to the number of active topics, then it is unlikely that the WebSocket server would be overwhelmed with the amount of messages. Precondition is that each WebSocket server subscribes to only those topics that its clients are currently interested in. This in turn requires that messages are published on sufficiently fine-grained topics. While databases like Redis and Postgres provide basic PubSub systems, a system with more difficult requirements will likely look at Kafka instead.
    – amon
    Mar 19, 2022 at 10:21
  • 1
    One approach would be to shard the channels/chat rooms: every user in a particular chat room connects to a particular WebSocket server, which does not send the messages to other WebSocket servers. If even one chat room is still too big for one WebSocket server, you could cluster several together, with a PubSub server that only handles traffic for that chat room (or a few chat rooms, up to the capacity of one PubSub server)
    – user253751
    Mar 21, 2022 at 9:42

2 Answers 2

1
+50

I think the biggest bits you need to solve for are understanding the individual nodes in your graph more deeply.

Load-balancers tend to regularly poll the related nodes to determine their load state.

Typically a load-balancer is a configurable piece of software. The configuration you provide it will determine when to start targeting other nodes in your configured resource stack. Whether they be containers, virtual machines, physical machines, or some other esoteric variant.

Those rules tie into the horizontal scaling. Typically the scaling rules of the system tend to go hand-in-hand with the load-balancer. After all, you'd need them to be similar, or once the load-balancer says 'oh, this node is too busy', it won't have another node to forward the request to. Should that configuration occur, the existing nodes would be loaded until they flat-line.

From a logical standpoint, having every WebSocket server (WSS for short) queued up into every possible topic is impossible as it is constrained by the limits of the hardware.

The connection, or socket, received for a given WSS identifies the client as part of the original transaction. Traits from that client's identity indicate the specific client's interests, such as what topics to subscribe to.

Implementation Scenario 1

As a client makes a request, they are registered with a given WSS for that connection.

Most likely what you'd have is some form of broad-based push notification from the PubSub to the WSS layer to say 'Something event has occurred.' Only when the event is received from the PubSub, would the WSS look deeper. Then a targeted query to the PubSub could be crafted from there. You'd use the 'long lived' connections to comprehend what kind of query to write. Each long-lived connection represents a specific client. That client ties to the topics that are of interest.

For instance, let's say Client A sent a message. The WSS pointed to by 2-3 would push that to PubSub 3-4. The WSS pointed to by 4-5 would receive a broad-based 'Changed' event. WSS 4-5 would then ask the PubSub if anything changed for Client B's interests.

If PubSub 3-4 says no, you'd immediately end inquiry there.

Now let's step back. WSS 2-3 and 4-5 each support 500 clients. Client 1 sends a message. WSS 2-3 and 4-5 both receive their push notification of 'Something changed.' WSS 2-3 and 4-5 would send a request with the relevant client IDs.

The PubSub would send a response targeted to the relevant clients. Most likely to reduce the chattiness of the response, it would wrap multiple event notifications up into one. Why? If say 50 of the clients subscribed to the same topic, why would you repeat the same data 49 additional times?

After this, the relevant WSS sends the notifications to the appropriate clients.

Implementation Scenario 2

As a client makes a request, they are registered with a given WSS for that connection.

The WSS for that client sends a registration event to PubSub 3-4. PubSub 3-4 needs to keep track of all of these separate WSS/Client pairs. Every event sent to PubSub 3-4 needs to check those pairings and send events accordingly.

This puts a heavier impact on PubSub 3-4, and may even require you to have vertical scaling implemented for it.

Client A sends a message. WSS 2-3 sends the event to PubSub. PubSub interprets the message and additionally pulls in context related to what clients are subscribed. Then parses the list of WSS targets to receive the push notification and sends the notification to WSS 4-5

WSS 4-5 receives a push notification for the common topic, and forwards that to Client B.

Taking a step back, WSS 2-3 and 4-5 each support 500 clients. Client 1 from 2-3 sends a message. 50 clients subscribe to the same topic. PubSub 3-4 parses the interests and sends the relevant push notifications.

After this, the relevant WSS sends the notifications to the appropriate clients.

Comments and Follow-Up questions

Both implementation scenarios have their benefits and drawbacks.

Now a question for you. When the load drops, and a horizontal contraction occurs, how does the load-balancer reallocate the open sockets from the dropped WSS? How is that change propagated down the line?

4
  • This is great. I feel as if I need to compress this. Is it accurate to say that scenario 1 is about using a "broadcast" approach to all live WSS. Meanwhile scenario 2 is using the pubsub system more like a cache (fast db) to look up connections and after that send message to approriate topic? In short the difference is almost push vs pull?
    – Frankster
    Mar 21, 2022 at 20:21
  • Regarding your question, I guess I would treat the scale in event as a new "event" in the "broadcast scenario". WSS6 is going down (here are all remaining 48 live connections), send a subscribe event.
    – Frankster
    Mar 21, 2022 at 20:23
  • I think a simple answer to the final question would be for the client listener (Client A and Client B in this case) get notified if the connection is closed. If it is, it reestablishes the connection. Making things very simple. Depending on the stack there may not be a 'x is going down' event to capture. Mar 21, 2022 at 23:57
  • Meanwhile scenario 2 is using the pubsub system more like a cache (fast db) to look up connections and after that send message to approriate topic? In short the difference is almost push vs pull? What I was trying to say is there's not one solution. It depends on your specific needs. Those may change as you learn more about the specific demands of your clients. I wish it were possible to magically know the 'one solution to rule them all' for a given domain, but there sadly is not. Mar 21, 2022 at 23:59
1

One of options is to use one queue/topic per client. This means if there is a message to be delivered to a particular client, the message has to be routed to their dedicated stream. And, the client, is subscribed to that stream.

For example, if a client A talks to two other clients B and C, then every message from B and C will eventually end up into stream for A. A will read messages and show them in a UI appropriately.

The above approach will work fine for smaller chats - either one2one or small rooms. What if you have huge rooms with many more users? This is very typical system design interview question - design twitter. For these larger rooms, you would have a different architecture. Instead of taking every message and putting it into every client stream, you would use one stream and have a watermark for each client - that will allow clients to pull messages as needed.

The main reason for handling large rooms differently is the fact that many users won't be online at any given time, hence pushing messages to their streams is a bit too much work.

To summarize:

  • for smaller chats - you put every message to a dedicated stream for that user
  • for larger chats - you subscribe users to larger topics

To address this question "I imagine the PubSub server managing 100,000 requests per second through sharding. But I can't understand how every WebSocket Server then handles the subscriptions. Because being subscribed to all topics on all shards will that not flood each WebSocket Server?"

Most clients usually process nothing, so handling their connections is just a matter of memory (websocket+pub/sub client). If pubsub is the issue, you can always decouple websockets from pub sub.

At any given time, your system will know which client is connected to which websocket server. When your backend has a message to deliver for a specific client, the backend will lookup the address of the websocket server and call an api like Deliver(To, Message) and that will funnel the message to a particular client.

Overall, there are many ways to get this done. Please, ask questions for clarification.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.