Basically, the idea is to have a new thread pool with a few worker threads in it every time a user log onto the application (a new user session is created).

Pretty much all the web applications I've worked with in the past had fixed amount of thread pools based on functionalities, but almost always performance issues would emerge when the user base grew.

With dedicated thread pool for each user, they would experience less lagging and not worry about competing for worker threads in a shared pool.

Am I wrong thinking of this way? If so, what have I overlooked here?

  • So what do you imagine will happen if the pool contains too many threads? – Esben Skov Pedersen Oct 8 '19 at 16:49
  • @EsbenSkovPedersen the computer gnomes add more CPUs as fast as they can mine the silicon – Ewan Oct 8 '19 at 16:51
  • You would run out of threads pretty quickly as threads are a finite resource, or most definitely run into thread context switching. Usually threads are allocated on a per process level, not user. – Jon Raynor Oct 8 '19 at 18:20
  • @JonRaynor That's how typically the threadpools are used. However, with that structure, users submitting similar tasks would have to compete for threads, which could dramatically increase the wait time of other users. – Will Oct 8 '19 at 18:28

When you look at the most highly scalable web applications (or services) today, the problem wasn't ensuring enough threads that were available, but ensuring the whole system could handle the concurrent web connections required to service the endpoints.

The biggest advancements in concurrent requests/second on one installation came from non-blocking I/O. Essentially, the operating system has the means of letting the application when a new client as connected, and when new bytes arrive. A single thread can handle multiple thousands of asynchronous connections as long as the non-blocking I/O is fully supported by your system.

A system can have too many active threads at once. When that happens, the operating system is spending more time switching contexts between threads than executing work within the thread. Having a global thread pool with a finite number of threads to process asynchronous I/O with an asynchronous platform will help the efficiency of handling thousands of requests per second. It's probably best to have the total thread count a multiple of the cores on your machine (across all CPUs).

Another very common answer to scaling is to simply have more servers hosting your application or web service. Each server is in it's own process, many times on separate hardware. If there is a proper "shared nothing" architecture then you can simply scale out as much as your load balancer allows.

  • You beat me to it. I think "More threads are not necessarily the answer." is not really strong enough. Any time you have more threads than CPUs there's some overhead involved. It can be negligible or insignificant but in the case of scaling up to a larger user base, you need less threads, not more. – JimmyJames Oct 8 '19 at 17:09
  • 1
    Removed that sentence so the paragraph starts with "A system can have too many active threads at once" which is a stronger statement. – Berin Loritsch Oct 8 '19 at 20:06

To add to Berin's answer: Adding more threads to the application beyond the number of CPUs (you can have more than one CPU per core) will only help performance in the case that your application performance is not CPU-bound. Most web applications are IO-bound which makes adding threads a viable strategy in this space.

However, there's also thread switching overhead: at some point adding more threads will decrease your performance. You might be surprised by how few threads it takes to reach this point. Keep going beyond that point and you will eventually cripple the system. The upshot is that you need to limit the total number of threads. Most likely the reason you see this divided by functionalities is based on the observed needs for these different activities. If you allocate on user, it could be difficult to optimize the thread allocations. You'd need limit the number of users supported by a host. This might work better for your application, or it might not.

The better option if you want to improve your utilization ratios is non-blocking IO (NIO) which allows one thread to handle multiple user requests in IO-bound applications. In a nutshell: instead of waiting (blocking) on IO requests, it moves on to do what ever work is available. It does this without incurring the cost of thread switching at the system level. The result is that a vast increase the number of users that a single server can support.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.