i have a problem in my scalable Pub/Sub application that it can not take more than 30 RPS when big amount of sessions are open against it. first i will explain the application structure.
the application is a spring-boot embedded tomcat with 200 tomcat threads. it is like i said a Pub/Sub application with multiple instances behind a load balancer. when a user opens a connection against the application it gets sticky to specific instance with a cookie he receives back. the Pub\Sub pattern is obtained by redis that sends the request to the instance which holds the recipient session.
the load balancer is RR so the connections are spread evenly between the instances. when a single instance holds more than 3k connections things are starting to slow. the RPS i can send decreasing drastically from 3000 RPS i could send before connections to 30 RPS with them. i assumed first that the problem is too little tomcat threads available but it is not the case as i monitor and see 100+ free tomcat threads that are not is use per instance. i can not find the culprit, seems like it's something related to the amount of sessions open against the instance but i can not say why. BTW the protocol used to maintain the sessions is xhr-streaming