java threading model for scale up

Question

I need some advice on the scale-up issue. we have a java application currently it works as below

the current system is using the Thread Per Request Model.

each client connection (long-running and streaming data from us) will create a new thread in the thread pool and a java internal blocked queue. the disruptor work thread (step B) use the incoming message header to push the message into the assigned java internal queue. the thread in the thread pool wakes up when the message arrives at the queue and process the message (a lot of business logic in this thread, include decode the incoming message which is cpu costly), after that, the thread use emitter to publish the result back to the client. the issue here is

once we have more clients. the thread pool becomes unmanageable. in our case 20K user streaming data means 20K running threads+ 20K java internal queues. my question is:

how to change this design so we can scale up the system? we can't use any messaging system in-between. we only can allow changing the thing within the green box.

here is more detailed info about the system.

• Each client connection would receive update of around 4 to 25 messages per second. (message is byte format, very small size)
• All messages have to process in order based on per-user connection. That’s why we use the java internal queue to keep the incoming message order
• Step A (Queue Listener) is a single thread lib (we can’t change it ) and have to process the message sequentially. That’s why we put a ringbuff after the listener, so we can remove the message off from the queue ASAP.
• The server is very powerful. We have 24 cpu (6 core each) + 128G RAM
• Step C processes the message from the jave internal queue and send it to clients. The process takes 100 ms per message from end to end. this step also decodes the byte message into java object.
• We want to achieve 20K concurrent user online.
• There is no requirement to share data between each user connection
• List We could run multi-instance of java application on the same box. But if we keep using the per request thread module, the number of thread still would be an issue.

Answer 1

once we have more clients. the thread pool becomes unmanageable. in our case 20K user streaming data means 20K running threads+ 20K java internal queues. my question is:

What you are calling a thread pool doesn't sound like thread pool to me. From Wikipedia's definition:

By maintaining a pool of threads, the model increases performance and avoids latency in execution due to frequent creation and destruction of threads for short-lived tasks.[2] The number of available threads is tuned to the computing resources available to the program, such as a parallel task queue after completion of execution.

The whole idea of a thread pool is to avoid having 20k threads. Instead you want to have just enough to keep your cores busy. Having many threads will means you have to keep all of them in memory which adds to the overhead. It might also add to the time it takes to context switch. Since you stated that your tasks in step C do not have any blocking IO calls, you can roughly need one thread per core, that is 24x6 = 144 cores. Since you want to read messages for each connection in order, you should assign connections to threads. With that distribution in mind you can also reduce the number of queues to match the number of threads.

Now you have one worker shoveling messages in the ring buffer, as before. You can keep the logic in step B as it is as well, except that it also needs to make a decision which connection goes to which queue. In the simplest case the assignment is queue_no = connectoin_no % 24.

In step C you now have worker threads that are always active, as long as there is work to do in their queue and sleep otherwise. There is no context switching involved and every single thread can use 900 MB of memory.

All of this assumes that messages are fairly even distributed on connections. If you have 5 connections that make up 90% of the traffic and 2 of them happen to be on the same thread, you might run into situation where one thread is idle and another one can't keep up. This can be fixed, but really depends on the shape of the traffic.