How many threads should I use in my NIO server?

Question

I've built a Java NIO TCP server, it currently uses four threads. One ServerRunnable thread that uses a selector and three worker threads.

I've been looking around for some information about this, as I've read in the past that you should only have one thread per processor core. So that's that.

But that got me wondering recently, and after a little more research I came across this thread.

Where, in the comments on the accepted answer user Donal Fellows points out the following:

Have at most one CPU-bound thread per processor allocated to the application. IO-bound threads aren't a big problem (other than the memory they consume) and it's important to remember that apps can be restricted to only use a subset of the system's CPUs; after all, it's (usually) the user's/admin's computer and not the programmer's.

With that in mind, am I correct in thinking I can safely increase the number of selector threads and worker threads in my thread pool.

My server thread reads input, processes the data into JSONObjects and then pushes them to a queue. The worker threads then take the JSONObjects from the queue, checks what type of objects they are and then pushes them to the database. So there is very little computational work going on there. Is it safe enough for me to increase the number of threads here, as in use more ServerRunnable threads and more worker threads? Say to double the amount of each for example?

What do I need to think about when considering something like this?

Answer 1

You have no reason to assume how many CPU cores the users system has. You development machine might run on a 4-core CPU with nothing else to do, but it could also be moved to a single-core virtual machine or a 32-core high-end server.

For that reason you should not hard-code the number of CPUs.

In Java, you can use ThreadPoolExecutor's to delegate the thread management to the JVM. You usually pass smaller work packages to the executor in form of objects which implement Runnable and leave the decision which thread processes each runnable to the thread pool. In your case, the individual JSONObjects would be such work packages.

The ThreadPoolExecutor allows you to set the minimum and maximum number of CPUs it is allowed to use. I would recommend you to make this configurable and default to Runtime.getRuntime().availableProcessors() when no configuration option is provided.

Whether or not your IO thread counts as an actual thread in regards to its CPU load depends on what it is actually doing. When it receives a high IO bandwidth and has considerable parsing work to do, it might. But we can not tell you this without profiling your application under real-world conditions.

Answer 2

Processing power is the most scarce resource in a computer. As I write this, the most advanced CPU I am aware of has 18 cores and costs around 6 grands. That limits your number of threads to 18 if you want to have true parallelism. Anything more than that is overkill unless you are writing GUIs or other latency-insensitive applications.

A single-thread is capable of handling at least ten thousand connections. Now imagine if you wanted to have 10 thousand threads, one for each connection?

Take a look on this article about high-availability servers with CoralReactor to understand how a single thread can handle thousands of connections through a demultiplexer and multiplexer.

Disclaimer: I am one of the developers of CoralReactor.

Answer 3

One of the things that's critical in designing a multithreaded system is figuring out how to split the work up in a way that keeps as many cores as busy as possible. Your proposed design uses two types of threads which are mostly I/O bound and puts some of the computational work on each.

I'd suggest that you make the I/O-bound threads separate and put the compute-intensive stuff in the middle using three kinds of threads:

• Input - Reads the input, puts it on an input queue and goes back to waiting for more input. I'm going on the assumption that the input is a single stream where it isn't practical to have multiple readers. If multiples work, it's safe to add more of these threads, up to the number of channels your input environment provides. The important point here is that offloading the input onto a queue as quickly as possible gets the thread back to reading more input or being I/O bound without adding any delay while it's processed, maximizing the ingestion rate.

• Processing - Grabs items from the input queue, converts them to JSON objects, decides what type they are, cooks up the correct database action and puts that on an output queue. In other words, this is all of the parallelizable, CPU-intensive work.

• Output - Takes items off the output queue and does your database writes. These threads do almost no processing and, like the input threads, spend most of their time waiting for I/Os to complete.

The benefit in this model is that you can use the state of the queues to make decisions about how to tune the system:

• Continued growth in the average input queue length means there aren't enough processing threads to handle the input load. The solution is to either add more processing threads or, if you're out of physical cores, get a machine with more. (Better might be another machine to pick up some of the load, but that's another discussion.)

• Growth in the output queue means you have a bottleneck in getting database writes done. Usually you'll want as many output threads as the database can handle in parallel. Any more and you're just offloading the overqueueing problem to the database. (Some databases are better than others at dealing with that, so whether you go for more output threads is a decision only you can make.)