How to handle thousands of messages to C socket?

Question

I currently have a setup looking like this:

 __________________      _________      ________      ___________
| Front end server |----| Varnish |----| NodeJS |----| C-service |
 ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾      ‾‾‾‾‾‾‾‾‾      ‾‾‾‾‾‾‾‾      ‾‾‾‾‾‾‾‾‾‾‾

NodeJS and C-service runs on the same virtual server.

The front end server may have hundreds of clients connected - each of these can generate hundreds of concurrent requests to the C-service, so the resulting requests to the C-service will number in the thousands.

From previous experience I know that sockets can only handle up to some limit of waiting connections (backlog). To handle that I implemented a message queue in NodeJS (it also converts HTTP messages into messages that the C-service understands, but that's a minor thing I could be doing in the C-service instead).

Problem:

• Varnish handles the thousands of requests easily
• The C-service can keep up with the thousands of requests, as long as they are queued.
• NodeJS... Not so much. (I tried clustering, but due to other implementation issues this is not an option.)

In other words, I have a NodeJS server that is severely limiting my throughput, but I don't know how else to handle that many concurrect requests.

I've been looking into message brokers such as Kafka/RabbitMQ/ZeroMQ, but they don't seem to handle my basic problem of connecting an HTTP service (Varnish) to an underlying TCP/Unix domain socket with message queuing. I'm not sure they're well suited for synchronous operation either.

Is this the wrong approach? What else could I be doing?

I need the responses to be sent synchronously, but not necessarily in the same order as they are sent from Varnish (I can run multiple instances of the C-service for better CPU utilization in which case sending the responses in order would slow things down).

Answer 1

Seems I was indeed looking for the wrong tool for the job. Well, maybe message brokers can do the job, but there is an easier solution: HAProxy.

HAProxy has a built in max connections settings for it's backend servers and will queue messages until the backend server can respond. It also supports connecting to Unix domain sockets (since version 1.5).

In the backend server definition add maxconn [limit], for example:

backend backend-server
    server backend 127.0.0.1:8080 maxconn 5

To support thousands of incoming requests it looks like it's necessary to add maxconn in both the global and frontend sections of the config, for example:

global
    maxconn 10000

frontend http-in
    bind *:80
    maxconn 10000

Answer 2

On the C side, you might consider using some message library, like 0mq.

Perhaps you could be interested by JSONRPC or some JSON libraries like jansson (probably to be used above 0mq...), since node.js is likely to be JSON friendly.