the current system is using the Thread Per Request Model.
each client connection (long-running and streaming data from us) will create a new thread in the thread pool and a java internal blocked queue. the disruptor work thread (step B) use the incoming message header to push the message into the assigned java internal queue. the thread in the thread pool wakes up when the message arrives at the queue and process the message (a lot of business logic in this thread, include decode the incoming message which is cpu costly), after that, the thread use emitter to publish the result back to the client. the issue here is
once we have more clients. the thread pool becomes unmanageable. in our case 20K user streaming data means 20K running threads+ 20K java internal queues. my question is:
how to change this design so we can scale up the system? we can't use any messaging system in-between. we only can allow changing the thing within the green box.
here is more detailed info about the system.
- Each client connection would receive update of around 4 to 25 messages per second. (message is byte format, very small size)
- All messages have to process in order based on per-user connection. That’s why we use the java internal queue to keep the incoming message order
- Step A (Queue Listener) is a single thread lib (we can’t change it ) and have to process the message sequentially. That’s why we put a ringbuff after the listener, so we can remove the message off from the queue ASAP.
- The server is very powerful. We have 24 cpu (6 core each) + 128G RAM
- Step C processes the message from the jave internal queue and send it to clients. The process takes 100 ms per message from end to end. this step also decodes the byte message into java object.
- We want to achieve 20K concurrent user online.
- There is no requirement to share data between each user connection
- List We could run multi-instance of java application on the same box. But if we keep using the per request thread module, the number of thread still would be an issue.