How many threads is too many to maintain a chat server?

Question

I'm writing a chat server and client. For the chat server I have 3 "services" all running on their own threads:

• ConnectionListener that loops over accept, and adds any new clients to a ConcurrentLinkedQueue
• MessageListener that waits for incoming messages from clients; also adding them to another ConcurrentLinkedQueue
• MessageBroadcaster that sits on the other end of MessageListener's queue, and sends out any received messages to the clients in ConnectionListener's queue.

I think that's all the background tasks that I'll need, but say in the future, I have another task that I think should operate concurrently in the background. At that point, the server will be using 4 threads for 4 separate background tasks.

Is there a point where I should reconsider my design? What is a better way of arranging this?

My question stems from 2 concerns:

• I hear threads are expensive. While that isn't an issue yet, I don't want to get into the habit of spawning a new thread for everything if there's a better way.
• I'm not sure if I should sacrifice "separation of concerns" to save using threads. In theory, the message receive/broadcast jobs could be combined; although they're separate duties.

Answer 1

I hear threads are expensive. While that isn't an issue yet, I don't want to get into the habit of spawning a new thread for everything if there's a better way.

Threads have overhead. The longer a thread lives, the less of a concern the overhead is. For example, let us assume a thread has a ton of overhead and requires 100ms to spin up. However, the thread lives for an hour. That is a tiny fraction of its life, so who cares? If you can bury the thread initialization in application startup when users expect delays anyway, all the better. If the thread lived for less than a second, then the overhead is a bit much and maybe you should consider alternatives.

That being said, do not spawn threads to do nothing. Each thread should have a purpose.

I'm not sure if I should sacrifice "separation of concerns" to save using threads. In theory, the message receive/broadcast jobs could be combined; although they're separate duties.

In my experience, multithreaded applications need multiple threads. Combining concerns with e.g. a listener and a broadcaster in this case will increase complexity to the point of it being unmanageable. It will create bugs. The key here is where do the threads block? If you have a listener that blocks on accepting a socket, it makes no sense for it also to handle requests. One of its concerns will suffer: requests will be delayed, or new connections might fail. Just create more threads.

---

Note 1: you should be using thread primitives, not raw threads. Since you tagged this question java I will point to the specific class: ExecutorService. The Executors class provides several factory methods to make it easy to create them for various tasks. The basic idea is you submit tasks (Future or Runnable) and let the framework manage the threads.

This has two primary advantages:

1. Managing threads can be hard. While it appears easy, there is a lot of boilerplate logic that is easy to screw up, resulting in weird, difficult to replicate (and fix) bugs. Push this responsibility into a robust framework that is well-tested and used in millions of other programs. If there were bugs then Oracle (or Microsoft for .NET, or whatever other vendor) would have found them by now.

2. Your program's purpose is not "to manage threads" it is "to run a chat server." Let the framework handle the threading concerns, and focus on the chat server aspect. By focusing on the tasks and not the threads, your code is more concise, expressive, and clear.

I wrote up an answer to this old Stack Overflow question that shows a use case for this. I do not want to derail the focus on this answer too much, so you can read more here as well as search both Programmers and SO: Creating an unknown amount of threads in Java?

---

Note 2: given that this appears to be a desktop/server application as opposed to embedded, there is not much need to be concerned with spawning threads. Modern CPUs (amd64, Intel Core) have multiple cores on-die, typically between two and six. Some models are hyperthreaded or superscalar, adding more logical cores that can execute code concurrently. Add in time slicing with today's high clock speeds, and there is no need to concern oneself with spawning too many threads as long as the threads are reasonable. In the case of your question, the threads appear to be performing small bits of work and possibly blocking on I/O. You are not spawning a million threads for digital video encoding, for example.

Even on embedded architectures, speed and concurrency are not as strong as desktop and server CPUs but should have no problem handling a few threads. My smart phone and tablet both have many applications running in the background and have zero performance problems even with many applications and threads running.

---

Note 3: if you are interested in learning more about the tradeoffs of a single-threaded algorithm vs. splitting it into pieces and using concurrency, there are a few topics worth exploring. Given the problem described in your question this is not an immediate need, but could make for some interesting and educational reading.

• MapReduce is an algorithm for splitting a task into pieces that do not rely on each other, performing those tasks, and joining the results together.

• Parallelism in databases involves multiple threads querying data and joining the results. There has been a lot of research into the tradeoffs of using multiple threads for a single database query and a lot of information is out there. Outside of schema/query optimization this is very useful information from a more theoretical perspective.

Answer 2

4 threads are nothing, especially since these background tasks that you described are mostly doing nothing but waiting for something to happen.

A modern server can easily have hundreds of such threads simultaneously running, precisely because the word "running" here is mostly a figure of speech: they are not really running, they are just waiting.

I open up Sysinternals Process Explorer on my desktop computer right now, select "System Information", and it says that I have about 957 threads running. Does that answer your question?

The problems start when your threads actually have processing to do, but this is actually quite rare. You might be writing lots of code and you might be thinking that this thread of yours is going to have lots of stuff to do, but in most cases this is just an illusion.

The vast majority of threads written by the vast majority of programmers spend the vast majority of their time waiting for something external to the thread to happen. When we say that a thread is waiting, we mean that it is sitting in an efficient wait state, and consuming zero CPU.

Examples:

• When a thread opens a file to read or write data from/into it, it is mostly waiting for the data to be transferred from/to the disk. While the data are being transferred, the thread is sitting around doing nothing.

• When a thread issues a SELECT statement to a database, it is mostly waiting for the database to execute the command, and then for data to arrive via the network. During that time, the thread is sitting around doing nothing.

• When a thread performs a network operation like reading from a socket, --you guessed it, it is just sitting around doing nothing.

So, unless you do something silly, like spin-looping, (also known as busy-waiting,) all of the code that you have written which constitutes your thread represents but a minuscule percentage of the total running time of that thread. Over the course of an hour it will usually wait for almost entirely an hour, and it will do actual work for a grand total of a few milliseconds.

In other words, the thread sleeps for aeons of computer time, and once every few centuries it wakes up to eat a quick breakfast and immediately go back to sleep.

So, I would not worry about these 4 threads of yours at all.