How to best approach planning and budgeting of software performance optimization efforts?

Question

There is a software system that has been with the client for some time now. If is feature-rich, mature, complex enterprise application that client uses in production. Due to the rapid expansion of the client's business the software system in subject cannot handle the volumes efficiently and overall performance does not satisfy the customer anymore.

So the customer asks the vendor of the software system for improvements in performance and efficiency of the system. The customer also expects the vendor to prepare a plan and ballpark budget so the cost of such optimizations is roughly known upfront.

What do you think is the best approach for such plan and high level budget that one could prepare for the customer? I'm asking the question because I've seen some of this kind of plans before and they are usually very vague and imprecise mainly due to the fact that such exercise is kind of chicken and egg dilemma. In order to know what exactly should be optimized and tuned in complex system the benchmarking, profiling and research is required. And that investigation is also the part of the planned project, so it is impossible to know upfront what efforts can such benchmarking results require to implement improvements.

Answer 1

I would be honest with them. If they plan to fund this, and it indeed will take a lot of time to find the performance issues, tell them it should be funded in two stages. First scope out the work needed to perform investigation, benchmarking, and analysis. The end result of that effort can be a proposal for what areas need to be optimized. Find where the most bang-for-the-buck is. This might be optimizing a function or refactoring/changing the system's architecture. You can evaluate the risk/reward and decide how much risk you/they are willing to take for how much performance gain.

If they're not willing to pay for the first phase, you may have to fund it yourself as part of doing proposal/analysis type work.

Then you can be better prepared to give them options for phase 2 -- actually optimizing the performance. You'll have sound recommendations and be able to actually provide some kind of estimates.

Answer 2

I can only give you a bit of personal experience. We had a product with a serious performance problem, and people could only hold their finger in the wind and guess what the problem might be.

I've done a fair amount of performance tuning over the years, so I asked the manager to let me try. I pulled a number out of the air - 4 weeks.

He was skeptical and said - won't it take you that long just to set up profilers and whatnot? I said I don't use profilers, I just take samples manually. He said how are you going to get enough samples to get valid measurements? I said if you have an infinite loop how many samples do you need? If it's not infinite, just long-running, how many do you need? He said ah-hah, and gave me the go-ahead.

It took me 2 weeks just to get the build/debug environments set up on my machine. There were two languages (possibly three, I forget). That got me to the point where I could reproduce the problem and take a few manual stack samples, and immediately pinpointed the first problem. From taking first samples to locating the problem took less than an hour.

Now comes the hard part - getting the owner(s) of the code to change it. The first problem was not too hard. The code was fixed, and there was a big reduction in execution time. Then I took some more samples, and located the second big problem. This time, it was not so easy to get the cooperation of the code's owner. The required change ran against his notion of "proper design". It seems my "political capital" was used up. So we did not succeed as far as we could have. If you can keep going over multiple steps, larger speedups are possible, but there is always the danger of the "blame game". That's where the owner of the code doesn't want to change it, because someone might ask them why they didn't make the code that way in the first place. It's hard to convey that performance problems are just like bugs - anyone can make them, and a measure of adulthood is being able to admit it.

Bottom line - about 3 weeks was spent, out of the 4 estimated. We got some of the possible speedup.

Lesson learned - If you're going to do this, it needs to be spelled out to all the developers that they may have to make multiple code changes, including design changes. What's more, the changes may appear "wasteful" in the sense of making prior changes irrelevant. You need to get their buy-in.

Answer 3

First you have to identify the performance blocks before you can estimate the time to fix. If the block is caused by needing a newer server with more memory and processing power that doesn't take long to fix. If they have a good equipment, but the problem is the basic design of the database, that could take a year or more to fix. Or maybe they need database partitioning to be able to access things faster.

In a mature product, you should have dbas finding the long-running queries and fixing them on a regular basis. Performance tuning should be continual. Further your devs probaly have a good idea of where the technical debt lies and that is a good place to start looking for performance problems.

I would estimate the time to identify the performance blocks and tell them that you will prepare an estimate to solve the problems once the problems are known. I would also start right now to look at your long running queries and fix them. Each incremental improvement will help keep the client on your side. You might also have them give you a prioritized list of the areas where they have the slowest performance to give you an idea of what to start with. You might even consider proposing an interative process when you spend a week or two identifying problems in one specific part of the application and then another week or two fixing those problems then move on to the next section, etc. That way, you can start improving before identifying all the items causing the poor performance.

There are known performance issues with certain types of SQL structures (this varies by database backend), but for SQL server, if you are frequently using such things as correlated subqueries, cursors, functions in joins and where clauses, scalar UDFs, views that call views that call views, then it is likely your database needs a major overhaul. If all your foreign keys are not indexed that is an easy, cheap (in terms of time) win. If your devs don't know what sargable means, then likely you have badly written queries and almost every query will need to be examined for poor practices.

Frankly, it is nice your client is willing to pony up some of the money for this, but you did the poor design, your company should pay for the majority of the fixes.

Finally, make sure you learn from this. If the problem is bad design of databases or poorly performing queries, train your devs to write performant queries and/or hire a data specialist. Anyone who designs an Enterprise system without data specialists in the design phase is likely to have a system that fails when the load gets high. Performance should be considered in the design phase of database. This is NOT premature optimization, this is avoiding known performance issues. It is far harder to fix a mature Enterprise system in production that to fix a bad design.

I'm not saying the database is the only source of your performance issues, just that it is one of the most likely places. You do need to profile to see what is happening. I can remember one time when we were having a performance issue and it turned out the application was calling the same proc thousands of times for each page when it only needed to be called once. But as a data specialist, my experience has been that there are many poorly designed and poorly performing databases out there.