Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
I'm thinking of how we do JVM memory monitor in a low overhead way in production environment even under busy hour.
Suppose I have two tomcat app server in production, load balance set up behind them. If I can see the jvm memory statistics I can tell load balance to stop sending the request to the server which will encounter OOM issue. Do this make sense? Jconsole or VisualVM eats more performance resourse is not my choice.
JMX would be the answer (Jolokia being a JMX interface).
You might want to also look at - https://stackoverflow.com/questions/242958/best-tools-to-monitor-tomcat
Others have provided suggestions on how to monitor memory usage ...
Suppose I have two tomcat app server in production, load balance set up behind them. If I can see the jvm memory statistics I can tell load balance to stop sending the request to the server which will encounter OOM issue. Do this make sense?
Sort of. But it isn't necessarily the best way to solve your problem.
Lets backtrack to the root of the problem ... the OOMEs. In the context of Tomcat, OOMEs are likely to be caused by one of the following:
To solve your problem, you first need to find out which of these is happening ... because the solution is different for each of them.
1) To see if this is a memory leak, you need to use a memory analysis tool to examine the long-term memory usage patterns. This will probably show a sawtooth pattern ... which is normal. What you need to look for is the the level of the bottoms of the "teeth" trending upwards over time. That indicates that something is creating garbage that cannot be collected; i.e. a memory leak.
If you have a memory leak, then the best solution is to figure out what part of your code is responsible, and fix it. Anything else ... including load balancing ... is a bandaid solution and could lead to worse problems down the track.
2) Having eliminated memory leaks, you need to figure out if the problem is that you are processing too many requests at once. I'm not sure of the best way to do that, but if this is the problem (or you suspect it is) then there are a few possible solutions:
Adjust the Tomcat server configuration to reduce the number of worker threads.
If your requests are I/O bound, then another possibility would be to look at the asynchronous request handling support available in recent versions of the Servlet spec - see http://docs.oracle.com/javaee/7/tutorial/doc/servlets012.htm. But that will be more work.
3) If the problem turns out to be that certain requests are using too much memory, then you need to figure out how to detect those requests before hand and "deal with them". Both detecting and dealing with these requests could be difficult ... and it is hard to advise without details of your application. But a couple of pragmatic solutions are:
Forward the anomalous requests to another server with a big heap ... where the OOMEs won't interfere with "normal" requests.
Increase the heap size. If you have enough physical memory, running with a larger heap could actually make your Tomcat servers more efficient ... as well as avoiding OOMEs.
In summary, rather trying to load balance to avoid OOMEs, I suggest you figure out why you are getting OOMEs ... and try to deal with the cause of the OOMEs directly.
Maybe jvmtop is worth a look for you.
It shows you in a "top-like" manner on a per-jvm basis monitoring metrics like memory consumption, cpu utilization, thread counts etc.
