I need some advice on the scale-up issue. we have a java application currently it works as below enter image description here

the current system is using the Thread Per Request Model.

each client connection (long-running and streaming data from us) will create a new thread in the thread pool and a java internal blocked queue. the disruptor work thread (step B) use the incoming message header to push the message into the assigned java internal queue. the thread in the thread pool wakes up when the message arrives at the queue and process the message (a lot of business logic in this thread, include decode the incoming message which is cpu costly), after that, the thread use emitter to publish the result back to the client. the issue here is

once we have more clients. the thread pool becomes unmanageable. in our case 20K user streaming data means 20K running threads+ 20K java internal queues. my question is:

how to change this design so we can scale up the system? we can't use any messaging system in-between. we only can allow changing the thing within the green box.

here is more detailed info about the system.

  • Each client connection would receive update of around 4 to 25 messages per second. (message is byte format, very small size)
  • All messages have to process in order based on per-user connection. That’s why we use the java internal queue to keep the incoming message order
  • Step A (Queue Listener) is a single thread lib (we can’t change it ) and have to process the message sequentially. That’s why we put a ringbuff after the listener, so we can remove the message off from the queue ASAP.
  • The server is very powerful. We have 24 cpu (6 core each) + 128G RAM
  • Step C processes the message from the jave internal queue and send it to clients. The process takes 100 ms per message from end to end. this step also decodes the byte message into java object.
  • We want to achieve 20K concurrent user online.
  • There is no requirement to share data between each user connection
  • List We could run multi-instance of java application on the same box. But if we keep using the per request thread module, the number of thread still would be an issue.
  • What do errors in your diagram represent? I find it confusing to have arrows going from a component to an actor. – Helena Apr 11 at 19:58
  • the system works as expected, but we can't scale up to support a large amount the clients due to each client currently has one thread to serve them. The actor is our client to request streaming live data from us. our system is a data distribution system. – london tom Apr 11 at 20:39
  • Sorry that was hasty typing, I meant to ask "what do arrows" represent? I guess it is flow of data? I usually read/write diagrams the other way around with the user on the left and data sources on the right. But I guess that is just a different convention. Though I think your diagram would benefit from labeling the arrows or having a legend. – Helena Apr 11 at 20:42
  • Also: How are messages dispatched in Step B? Is there usually one message per user, or all messages go to all users? Or every message goes to different sets of users? – Helena Apr 11 at 20:45
  • Too little info for me. Profiling seems to have been done. Load testing with varying dummy data hopefully too. Sometimes one can change the granularity of concurrency: larger/smaller pieces per thread, or defer some costly operations to a later point. There are others more experienced than me. – Joop Eggen Apr 11 at 20:46

once we have more clients. the thread pool becomes unmanageable. in our case 20K user streaming data means 20K running threads+ 20K java internal queues. my question is:

What you are calling a thread pool doesn't sound like thread pool to me. From Wikipedia's definition:

By maintaining a pool of threads, the model increases performance and avoids latency in execution due to frequent creation and destruction of threads for short-lived tasks.[2] The number of available threads is tuned to the computing resources available to the program, such as a parallel task queue after completion of execution.

The whole idea of a thread pool is to avoid having 20k threads. Instead you want to have just enough to keep your cores busy. Having many threads will means you have to keep all of them in memory which adds to the overhead. It might also add to the time it takes to context switch. Since you stated that your tasks in step C do not have any blocking IO calls, you can roughly need one thread per core, that is 24x6 = 144 cores. Since you want to read messages for each connection in order, you should assign connections to threads. With that distribution in mind you can also reduce the number of queues to match the number of threads.

Now you have one worker shoveling messages in the ring buffer, as before. You can keep the logic in step B as it is as well, except that it also needs to make a decision which connection goes to which queue. In the simplest case the assignment is queue_no = connectoin_no % 24.

In step C you now have worker threads that are always active, as long as there is work to do in their queue and sleep otherwise. There is no context switching involved and every single thread can use 900 MB of memory.

All of this assumes that messages are fairly even distributed on connections. If you have 5 connections that make up 90% of the traffic and 2 of them happen to be on the same thread, you might run into situation where one thread is idle and another one can't keep up. This can be fixed, but really depends on the shape of the traffic.

  • thanks for your suggestion. I really thought about your design. as you suggested we could group our connections and we will have much few queue and thread. but as you said in your last paragraph,we will face the issue that some connection could be mush busy than others. any suggestion to address this issue? – london tom Apr 18 at 20:03
  • @londontom I said you might have this issue, but I cannot tell since I don't have any data about your real distribution. Any fix for that would really depend on the actual distribution since there is no one-size-fit-all approach. – Helena Apr 18 at 20:05
  • 1
    If your connections don't differt that much (like only by factor 10) then your will most likely not have this problem, since the law of big numbers will even things out for you. – Helena Apr 18 at 20:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.