Load Balancing of requests coming from a single client

Question

I have a question regarding load balancing of a service. My load balancing requirement is as follows:

1. I will have multiple application servers for a non-http service. Application servers will be behind a load balancer(layer 4).
2. A client will initiate a connection to the load balancer and will send multiple requests over this same connection.
3. Load balancer need to distribute the requests from this single client connection to the multiple application servers in the backend.

Will round robin load balancing strategy work in this case? Does round robin load balancing strategy distribute only connection requests from client or it distributes the work load requests also?

Please suggest any load balancers which support such load balancing requirement?

Answer 1

Level 4 load balancers won't help you here. These can only distribute load based on transport information, not based on application protocol. In particular it is not aware that your single connection is composed of multiple requests and can be split. Especially if that is a custom, non-HTTP protocol. It won't do that, it will simply forward that entire connection to one of the target servers.

That being said, the above is not always true. It is sometimes possible. For example if your custom protocol lives on top of QUIC (which is considered a transport protocol, even though it is a protocol on top of UDP), then there is enough information to distribute requests (assuming one request per one QUIC stream). One of the reasons why QUIC is so cool. And it seems that at least HAProxy does support such QUIC load balancing, although recompilation might be required: https://www.haproxy.com/blog/how-to-enable-quic-load-balancing-on-haproxy

But generally you need an application load balancer. And since you don't use HTTP, likely a custom one.

Or you don't. Often transport load balancing is enough even in such case. As long as you have big enough traffic, i.e. enough parallel clients. In such situation it likely won't matter much. Except that you say "single client", which is a slightly weird case.

As for strategies roundrobin is very simple and in most cases good enough.

Answer 2

Roundrobin will happen by default in HAProxy unless you have sticky sessions enabled. Roundrobin will load balance your single client to multiple backends. I imagine it is the same with NGINX, but my experience is mostly with HAProxy. Yes roundrobin is what you want.

I missed the single connection requirement so the answer above is just wrong since HAProxy will only do either TCP load balancing or HTTP load balancing.

Interesting problem and I like the other answer, but I am not able to comment on it.

To add to their answer, custom work on the server sounds like what you need (without knowing more of the entire goal).

I would probably use a quick API like fastapi and just accept job details in a POST or if the data is too large either just write a socket server in python or use sshkeys and scp to upload the job and write a python watcher of the target directory.

You could then write logic around distributing the workload to other servers. Round robin or even just random for the single connection would probably work. But we need more details about your problem and goals.