How is server A faster than server B in some (or all) cases if B employs more parallel tasks?

Question

After researching modern web servers, I feel that I am pretty ignorant when it comes to server design and concurrent task management by both the operating system and the server. I understand some important basic functionality such as thread/process blocking and that the operating system will context switch from a blocked thread to a ready thread. However, I don't think that I have a good enough intuitive understanding of the overhead costs of various designs under the multiprocessing paradigm.

How exactly can a server, or any program for that matter, which processes requests in parallel across all available cores/processors possibly be slower than a server which does so on only one or a few? One cost I am aware of so far is that reading/writing the amount of memory needed for each parallel task is more costly than for one. I understand this point, but then in which cases does this cost not outweigh the added performance of running a queued task in parallel?

I understand server architecture can be very detailed and complicated, but any fundamental understanding that's offered would be very helpful. This question grew from me not being able to figure out how Node.js, which doesn't max out core utilization, can be faster than Nginx (which I have read does). Both have "asynchronous IO", does node just do it better?

Answer 1

There are several factors involved. Mainly, it's not immediate to determine what are we measuring and to what purpose.

For example a system serving one request in 12 ms is slower than a system serving one request in 10 ms...

...but the first system can serve eight requests in parallel, so it can dispatch eighty requests in the time the second can only dispatch twelve. The time for a single request is longer, but the time for a navigating set of requests is shorter.

Next, serving a resource isn't necessarily always the same thing. A system able to do efficient in-memory caching could take 10 ms to serve the first instance, and then 1 ms to serve all the others. Then the caching, single-process server dispatches one hundred eighteen requests instead of twelve.

That's why the choice of a specific server (and architecture) must be at least in part influenced by what and how is requested, and why often there is not a single system behind a website, but several (CDN, static cache, multiplexer, application server, all with different levels of redundancy).

Answer 2

When people thing "parallel", they think "speed". Wrong.

The purpose of multi-threading in a server is mostly for simplicity of programming. It only saves time if the threads are CPU-bound and can exploit multiple cores.

Usually the threads are not CPU-bound, they are I/O bound. The simplicity comes from each of many threads watching one synchronous stream, rather than one thread trying to handle events on hundreds of asynchronous input streams.