-4

i have a problem in my scalable Pub/Sub application that it can not take more than 30 RPS when big amount of sessions are open against it. first i will explain the application structure.

the application is a spring-boot embedded tomcat with 200 tomcat threads. it is like i said a Pub/Sub application with multiple instances behind a load balancer. when a user opens a connection against the application it gets sticky to specific instance with a cookie he receives back. the Pub\Sub pattern is obtained by redis that sends the request to the instance which holds the recipient session.

the load balancer is RR so the connections are spread evenly between the instances. when a single instance holds more than 3k connections things are starting to slow. the RPS i can send decreasing drastically from 3000 RPS i could send before connections to 30 RPS with them. i assumed first that the problem is too little tomcat threads available but it is not the case as i monitor and see 100+ free tomcat threads that are not is use per instance. i can not find the culprit, seems like it's something related to the amount of sessions open against the instance but i can not say why. BTW the protocol used to maintain the sessions is xhr-streaming

1
  • 2
    What happens if you take the load balancer out of the equation and run a single instance? Do you still see the same relation between the number of requests that can be handles and the number of sessions? How does your Pub/Sub system work? How many publishers are there? How many Subscribers? How does this relate to sessions and requests? May 19, 2020 at 11:21

1 Answer 1

1

You have to do a lot of checks. Note that profiling and performance optimization is art of asking questions and finding answers. You could start from couple of them:

  1. Which resources are (almost) exhausted? You have checked just one: Tomcat threads, but what about RAM/CPU/HDD IO/network IO?
  2. Does all server instances use some shared resource, like reading from same DB or connection with load balancer? Do these resources work well?
  3. What part of request processing takes most time on low load? On high load?
  4. Do you see some idle time when instance is processing request on high load? (F.e., step1 takes 20 ms, step2 takes 20 ms but on higher level step1-and-step2 takes 50 ms. Or response from DB took time and app just waits it)

While you are finding answers, you'll probably ask more questions. Write them down and then find answers. If you answer to all your questions, then you probably want to create another post here and describe what investigations you have done. It will be much more specific question - and that's why you'll get more accurate answers.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.