Sometimes it comes to me that the biggest challenge for an engineer when he find his application getting worse in performance is lack of enough information.
Imagine that you go through the weekly performance report from your application's access log, and find you get much lower response times from July. All you can do is to pray for it not to get worse the next time.
Without more details on the type and nature of your application, it is hard to give more than general hints:
If you do not have sufficient information, you need to collect the information you need. If you do not have a hard goal for what is satisfactory performance, you need to get one (otherwise you cannot tell if you reached the goal or not):
Information is typically collected in several ways:
Continue refining your knowledge of the problem and find places to improve until you've reached your goal.
Can you run the software under a debugger or IDE?
Does the debugger have a "pause" button, or can you interrupt the program by typing Ctrl-C or some such keystroke?
If the program is getting slower and slower, it is doing at least one thing it doesn't need to be doing, and doing more and more of it. Since it could be running in, say, 1/3 of the time it is currently running, that means 2/3 of the time it's doing things it doesn't need to do. If you can take an X-Ray snapshot at a random point in time, chances are 2/3 you will see it doing the unnecessary thing(s).
Pause it while it's running, and examine what it's doing and why. Especially examine every line of code on the call stack. See if you can explain to yourself or someone else precisely why, in detail, that particular instant of time was being spent. (You don't need to measure anything. You need to see if what it is doing can be gotten rid of.)
Repeat this a few times, like 5 or 10 times.
If what the program is doing at that point in time is not really necessary, and it is doing it on more than one time you stopped it, you have found something you can fix that will give you a big speedup, guaranteed. The bigger the problem is, the quicker you will find it.
It's got nothing to do with requirements.
It's got nothing to do with measurements.
It's got everything to do with just "cleaning house", by this method.
Here's a fairly typical example.
Edit: It's conventional wisdom to hear "measure measure" or "use a profiler". What is not conventional is to hear how much speedup was achieved that way. The few times I've heard the results of profiling, it was like 10% to 40%, or a factor of 1.1 to 1.4. That's pretty anemic. If a series of problems is found and fixed, there is a compounding effect, as shown in the example above.
P.S. Here's an example in C++ of a 3 order of magnitude speedup, containing all source code versions, copies of samples, and blow-by-blow description of how it is done. Some programmers have learned/discovered how to do this, but most have not. It couldn't be simpler. To this day, I am still totally mystified that it is not common knowledge. The only explanation I can see is that teachers don't work with programs large enough to require this kind of tuning. What they do is teach gprof, for no other reason than that it's there, so they can teach it and move on. What that does is infect their students with all the incredibly persistent myths of performance tuning, resulting in exactly the problems you describe.
P.P.S. In case the point isn't clear, any thread, whether it is alone or among thousands, has a certain minimum amount of work it absolutely must do to accomplish its purpose. Anything it is doing beyond that is taking extra time. In the example linked to above (which is only one particular example - every app is different) these "bottlenecks" were removed:
push_backnew and deleteAdding up to over 99%! By removing them you get orders of magnitude speedup.
Now the kind of thing I hear is "Well sure, it's silly to do 2, and 6 wasn't necessary either." Hey, nolo contendere, but what about the other four "bottlenecks"? If you don't fix them, how much speedup do you get?
If you want serious speedup, whether or not a profiler is used, you have to clean out all of the problems ! Any ones missed will be the dominant speed limiters.
Although generating metrics is useful, I take the following approach:
1> Run perfmon/task manager (on win) of top (unix) to see if CPU, disk, network or memory are being used heavily/thrashing/flatlining and investigate as appropriate. Ideally disk/network/memory should not be using anywhere near all the available resources. CPU at 100% is ok where you are doing something CPU intensive. Check that you are not running more threads than CPU's (unless the application is light on CPU use and disk is not an issue).
2> Do an isolation test to see if there are competing processes on the hardware that are using resources that could affect performance.
3> Check to see if Concurrency (if applicable, eg web app) has increased. If it has you need to look at you application to find the bottle necks that need to be addressed. Usually these will be in the database (index and table locking issue, slow queries) but may be in the infrastructure set up (ie enough CPU, memory)
4> At this point you probably have a data volume issue. Now look to see what has changed. Are data files getting larger? are database tables getting larger? If data sizes are getting larger then you need to understand what the application is doing to determine what course of action to take. If you use a database, you should check that the appropriate indexes are used and that queries are optimized. A SQL profiler is handy here as well as visual inspection of the code (assuming you know or know someone competent in database optimization) You are looking for the longest running query and the most frequently run queries. Another option is to look at archiving or deleting old or irrelevant data, if possible. Also ask if all the data is really required to be processed/stored. You also need to look for locking situation, code inspection and SQL profilers will help.
5> If the application is CPU intensive, you may like to look at optimizing the CPU intensive portions of code. Visual inspection, trace statements while monitoring CPU usage and profiler tools will help identify code that could be changed. The solution may be to use a different algorithm of remove/reschedule processing tasks. A set of experience eyes may be of assistance here.
6> remove any needed debug/logging code. Remove anything that is not really needed to run.
Again, the key is to understand what and how the application is operating, and thinking about how these steps interact with the hardware/database.
Performance tuning is a critical skill in any application that hits a database. Every database has a large book written about how to perfromance tune. Get the book for your dabase backend, read it cover to cover and then start taking the steps you need to identify and resolve performance issues. If your database was designed by someone without this knowldge to begin with, likely you have a bad design that has badly designed queries as well and possibly poor indexing. In this case you might want to read about refactoring datbases as well as you will likely need structural changes as well as fixes to the indexes and queries. This is complex stuff, you can't learn it all from questions on a website like this one. Just praying it will get better is counterproductive, there are many things you can do to make it better. And yes as time goes one and more data is added the queries will get slower and slower especially if they are poorly designed.
There are also non-database reasons for poor performance, but get the database tuned properly and it is likely that many of your issues will go away.
Ah the old data growth issue...
I've seen this so many times:
All this can be avoided if a little thought is placed at the beginning of the project.
