How did you find the slow performance in your application?

Question

What makes the web application perform and scale better is always a big topic. And finding the performance problems and tuning them is another...

Here is some my thoughts of how to "finding" performance problems:

For a "new" api/application or other

• Analzying the detail api and then preparing the Jmeter/Grinder testing scripts for it.
• Using different load to identify the threshold for the api
• Adding profiling codes find the slownes
• Restart from point one again..

For a "old" api/application or other

• Analyzing the user pattern from the access detail log
• Simulate the real user load to find the slowness
• Adding profiling codes find the slownes
• Restart from point one again..

So,how can you identify the performance problems?

Answer 1

The key to tracking down performance problems is:

1. Know when they exist.
2. Have sufficient context to figure out what was going wrong when they were slow.

The key for both of those is logging. The ideal is to have logging with optional logging levels that will spew out more detail which can be selectively turned on.

For a very good example of what works, take a look at Oracle. At all times, as part of the basic functionality of the system, it keeps track of what queries were run, and how long they took. DBAs can go and look at the situation to figure out where performance is going. (Don't just look at slow things, in a system under load a very common fast query can be a bigger problem than an occasional slow one.) Furthermore you have the ability to take a query, run it, and have Oracle dump out a detailed trace of exactly what happened, and where time went. Based on those dumps it is possible for an experienced DBA to figure out exactly what happened and where the bottleneck is.

Yes, there is a constant overhead from having this monitoring present. They try to minimize it, but it is still there. However the first time that it helps you locate a performance bottleneck that you hadn't realized was there, it pays for itself in spades. Without monitoring you're praying that you don't make any silly performance problems. Prayer is simply not a reliable way to get to scalability.

If you have a complex system with lots of RPCs, life gets more complicated. The unfortunate reality is that tracking down a seemingly random slow front end request to an RPC several layers deep that may or may not fire can turn into a nightmare. The solution, which practically nobody does, is to have your RPC mechanism have the ability to label a small fraction of requests as "tracer bullets". Those requests, and all RPCs recursively through your system, will get logged in detail, and those logs collected together to give an accurate picture of those requests. Sure, there is overhead to doing so. Which is why something like 0.1% of requests are logged in that detail. But when the system has trouble, you can now go in and look for a slow traced request, open it up, and immediately see where the performance problem is.

Yes, this is a lot of work. Yes, it is invasive. But I cannot stress enough how important this is to have in a large complex system. As the old saying goes, failure to plan is planning to fail. If you don't have a plan to figure out performance problems, when they happen you won't have a way to figure them out.

Answer 2

I've also implemented production performance monitoring systems to help identify performance problems; the key, of course, is making sure your performance monitoring doesn't hinder performance itself!

But really, a simple performance monitor that (in a web application example) just logs how long it takes to complete a request and how long it takes to render a page is enough to focus your efforts on the requests or request types that are the slowest, or present the greatest load (avg. execution time * execution count), and therefor are good targets for optimization.. Then, using a test platform and the basic steps you outlined in your question, you can narrow down the cause of the performance problems and begin your optimizations.

Depending on your situation, it's also good to monitor any dependent processes individually: profile database performance, disk performance, and so on, to make sure that all of the moving parts of the whole system are working as smoothly as possible. A perfectly optimized application can be slowed to a crawl by a poor-performing database.

Answer 3

I prefer not to have performance monitoring in my production code. I try to have pretty good unit test coverage (every significant method has at least one unit test). Visual Studio automatically times how long it takes to perform each unit test. For a general overview, I just run all tests to get a general idea of where bottlenecks might be.

Answer 4

For a web application, timing individual requests is a good start. Firebug works well for that. If you combine that with Ironcode's suggestion to rely on unit tests, then you should be able to locate most of the bottlenecks.

Beyond that... it depends on your app. There are various tools that you could use for profiling various aspects of the code. For example, if you're using Hibernate, it lets you collect statistics on how long all the database queries take to complete.

Answer 5

Well, usually it is the users who complain the app is slow (-:

All humor aside, here are some things I've found to be helpful based on how the app will be used, by whom, and when:

• Identify what parts of the app are slow (existing app) or cannot be slow (new app). This can either come from users, your experience or performance requirements. Also identify what use cases are slow, which can help narrow the scope. E.g., if use cases that have no DB interaction are slow, you can rule out DB issues.

• Identify what times the app is slow. "Always slow" is different from "Slow on Mondays between 9am and 10am." Maybe there is a backup of the DB that runs at that time. Maybe there are 10x more users on the system at that time.

• Have your DBA run stats and reports for the times your app is slow.

• Have your sys admin run stats on the file system for Disk I/O, CPU, RAM, and other such resource consumption.

• Have your network admin run stats for network traffic between the various servers.

• Analyze logs. This can vary from enabling %T in Apache log format to setting log format to "DEBUG" (Note: Logging in Production environments is usually set to "ERROR", so best to do this in a lower environment like Test).

• If you are running a Java based application, take thread and heap dumps. Analyze them for locks or memory leaks. Memory Analysis Tool is a useful tool for heap dump analysis, and I believe it can be used as a plugin with Eclipse. Enable garbage collection logs, and analyze them too.

• If your organization can afford COTS profiling tools like OPNet's Panorama and ACE, or NewRelic's tools try them as well.

Sounds like you are aware of some of these concepts. I am submitting my answer for general audience who may or may not be familiar with these ideas.

Either way, HTH,

KM

Answer 6

As far as I know, in enterprise applications, there usualy are NFRs present stating the general effectiveness of system under test. By my project, it works like this:

-the customer specifies what he understands as "effective", "responsive", etc.

-we prepare example testdata (not "let's hope for the best" dataset, but "let's ride this app like there's no tommorrow" dataset), and we prepare usage scenarios to be run.

-then, we let it run and burn some CPU cycles, for like 48/72 hours, and gather general system statistics (that way we don't have to place specific and unrealistic monitoring code in the app).

-in the end it is possible to say whetever there is a specific problem (eg. the system should do 10k transactions per hour, but does only 2k - that's performance bottleneck somewhere), and we should be able to tell which usage scenario/data combination caused unwanted behavior. From now on, it is possible to mimic specific examples in specific benchmarks, and work on the solution.

If you need to test specifically a webapp (eg. interface responsiveness under stress), you may consider using distributed selenium (I believe, there are some companies offering such services, using their own datacenters)