application can not handle certain amount of RPS when 3k+ sessions connected

Question

i have a problem in my scalable Pub/Sub application that it can not take more than 30 RPS when big amount of sessions are open against it. first i will explain the application structure.

the application is a spring-boot embedded tomcat with 200 tomcat threads. it is like i said a Pub/Sub application with multiple instances behind a load balancer. when a user opens a connection against the application it gets sticky to specific instance with a cookie he receives back. the Pub\Sub pattern is obtained by redis that sends the request to the instance which holds the recipient session.

the load balancer is RR so the connections are spread evenly between the instances. when a single instance holds more than 3k connections things are starting to slow. the RPS i can send decreasing drastically from 3000 RPS i could send before connections to 30 RPS with them. i assumed first that the problem is too little tomcat threads available but it is not the case as i monitor and see 100+ free tomcat threads that are not is use per instance. i can not find the culprit, seems like it's something related to the amount of sessions open against the instance but i can not say why. BTW the protocol used to maintain the sessions is xhr-streaming

Answer 1

You have to do a lot of checks. Note that profiling and performance optimization is art of asking questions and finding answers. You could start from couple of them:

1. Which resources are (almost) exhausted? You have checked just one: Tomcat threads, but what about RAM/CPU/HDD IO/network IO?
2. Does all server instances use some shared resource, like reading from same DB or connection with load balancer? Do these resources work well?
3. What part of request processing takes most time on low load? On high load?
4. Do you see some idle time when instance is processing request on high load? (F.e., step1 takes 20 ms, step2 takes 20 ms but on higher level step1-and-step2 takes 50 ms. Or response from DB took time and app just waits it)

While you are finding answers, you'll probably ask more questions. Write them down and then find answers. If you answer to all your questions, then you probably want to create another post here and describe what investigations you have done. It will be much more specific question - and that's why you'll get more accurate answers.