0

At this link (https://blog.heroku.com/timeout-quickly#how-webservers-work), I read:

All webservers will work in a similar way. Any new request will go to a queue, and the server will process them one after the other.

This means if you have 30 requests in your queue, each taking 1 second to be processed, that will take 30 seconds for your server to empty the queue. If one of those requests is a file upload for example and takes 5 minutes to be processed, it means that any other request will be stuck for 5 minutes. That's 5 minutes during which no one else can visit your app.

This reasoning is used to justify having a 30 second timeout for every request on Heroku-hosted web servers (https://devcenter.heroku.com/articles/error-codes#h12-request-timeout).

However, is the above really true for Node, which is supposed to be async? Is there any real justification for having such a timeout for a Node server running on Heroku?

1

Yes and no. Many servers these days use non-blocking I/O, which allows the server to quickly switch between sockets as they provide more data. To be clear the concept of there being a queue of work is still accurate, but the server is notified when there is the next chunk of data to process.

When serving up static files, this works rather well, as the same server can handle thousands of connections at once. However, when you start dealing with server side processing, you have to wait until enough data comes in (typically at least the request headers) to begin handing logic over to your web application.

Usually, there is a worker queue here as well. The server has a certain number of threads available to actually do work. So let's take that queue of 30 requests again, with 1 second response time on average. If the work queue is 30 threads you would be working on the entire queue at once.

I'm not as familiar with Node's architecture specifically, but there are several ways to make asynchronous work to support hundreds of requests simultaneously. One such animal is the JavaScript Promise concept.

  • You set up a Promise so that work can happen in the background
  • Work is started while you are doing other things
  • As soon as you wait on that promise, the task gets moved out of the processing queue and other work can happen in it's place
  • Once the Promise is fulfilled, the wait is over and the task can come back and resume where it left off

I'm kind of glossing over a lot of details here, but the point I'm trying to make is that the description that the Heroku developer gave you is only one way of doing things. Node has a reputation to scale pretty well even on a modest machine, but it's not the only web server that does.

  • OK, so that is my point; it seems like there is no justification for the 30-second Heroku timeout, as far as Node is concerned. It is that they use the same routing infrastructure irrespective of the server technology being used? – prmph Jul 28 at 14:37
  • The bottle neck may be where the server hands over processing to your code. The description the Heroku documentation was intended to be "how the average web server does things" so they can show how awesome they are. Since it's marketing material, they are overstating and oversimplifying that description. There is a very real threshold or limit to how responsive a web server can be before you need another one with some load balancing in front. It just that threshold is going to be a little different for each server. – Berin Loritsch Jul 28 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.