0

Imagine a typical HTTP service that does async db queries. If HTTP requests are received more quickly than the db can complete queries (such as because the db disk or network is slow), the Promises will accumulate in the event loop and presumably cause memory pressure, trigger swap-hang etc. I would rather the HTTP service stop generating requests if the server cannot consume them fast enough.

As a possible solution I was thinking an "asyncLimit" property of the db context (or something else) could indicate that IncomingMessage.reader.pause() should be called. The db query would still be submitted anyway of course since you can't just drop it at that point. But after a while, pause()ing streams should throttle requests. After long enough, kernerl buffers will fill up, client buffers will fill up and clients will stop sending data. Then something may or may not call resume() depending on the state of things (DOS?).

So how would you deal with this scenario?

Of course any async calls could build-up in the event loop and cause problems so it would be nice if the solution was generic to the arguably very differnt async / await idiom. If async / await is thought of as like streaming Promises, the flow control of readable.pipe() could be inspiring.

  • A promise sitting there in memory is a very tiny piece of memory and does not affect your server in any meaningful way and node.js is particularly good and efficient at having lots of asynchronous operations in flight while most are waiting for something. If you overwhelm your server with too many simultaneous connections, you may have connection issues with too many open connections waiting for an answer, but that would be more of an OS and OS configuration issue than anything to do with node.js. – jfriend00 Nov 7 '19 at 3:16
  • If you completely control the client library that accesses your server, you can implement limits on how many simultaneous connections a given client makes to your server. Beyond that, all you can do is start immediately failing and disconnecting new incoming connections with some 5xx error code once your server reaches a predetermined limit. – jfriend00 Nov 7 '19 at 3:18
  • FYI, async/await is just promises so no real difference whether something is using await p to await a promise or p.then() to await a promise. Same impact on the server (just a small promise in memory). – jfriend00 Nov 7 '19 at 3:20
  • There is other data that is not insignificant. For each promise, there's an HTTP request and response object and everything that goes with that including any data that was decoded from it. There's no point in having that pile up in memory if you can throttle it like pipe() does with data. – squarewav Nov 7 '19 at 4:25
  • There's no http response and request object until the connection is actually being processed. If you're overwhelming your server, the incoming TCP connection is probably setting in an OS queue waiting for your server to give it some attention. As I said in other comments, the only way to "throttle" here is to immediately drop incoming connections when you're over some limit. In larger installations, that may be done by additional hardware in front of the server (such as a load balancer or other security hardware). – jfriend00 Nov 7 '19 at 4:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.