At What Point Should You Start To Think About Performance?

Question

As I'm building applications I find myself constantly asking if this is best way to perform or implement a certain functionality. Often, I'll post questions on stackoverflow or another forum desiring feedback only to receive comments about how not to "put the cart before the horse" regarding performance. Do most programmers really not think about performance until the application is finished, or performance is absolutely unacceptable?? I mean, I understand that development environments differ from production environments and that you shouldn't completely rely on the results from your dev laptop...but, there are practices and techniques that yield better performance than others.

Is it bad practice to consider performance throughout the development process? Should I push these considerations off until performance actually is tanking??

Update

Just to be clear, I'm talking about the situation where you are considering or just about to work on some piece of functionality. You know there are several ways to implement it, but you're not quite sure how well each implementation will scale. Also there might be several techniques you're not even familiar with. On a small scale any of the approaches would probably be adequate, but on a larger scale some will keep up and some won't. Often when I ask for opinions or guidance the response is: worry about it later...

Answer 1

Deferral of performance considerations is sometimes based on a misapplication of the phrase:

Premature optimization is the root of all evil.

If you read the complete quote, what Knuth was trying to say is that micro-optimizations applied during development without profiling are generally inadvisable, because they lead to less maintainable code without necessarily achieving substantial performance benefits.

But that doesn't mean you should not consider performance until the application is almost finished. If you do that, you may find that performance is inadequate, your design architecture doesn't support better performance, and you have to start over.

There are a number of things you can do during development to achieve good performance, without esoteric (and premature) optimizations:

1. Use a sensible, well thought-out architecture.
2. Use data structures properly.
3. Use technologies (libraries, frameworks) that perform adequately.

If you do these things, you will find that any performance optimization that needs to occur will be confined to a small part of your code. Profiling will identify that code, and allow you to focus your performance improvements where they will do the most good, without sacrificing maintainability.

Answer 2

Here's what NOT TO think about:

• Is ++i faster than i++?
• Is switch faster than if?
• Should I inline my functions?
• Are virtual functions slow?
• Is C++/Java/C# faster/slower than the other?
• blah, blah, ...

Here's what TO think about:

• What's a realistic expected workload?
• How often does the input information change, and who provides it? Does it make sense to consider precompiling?
• Have I kept my data structure as simple and as normalized as possible? That means not worrying about hashing and stuff like that.
• Have I kept notifications to the absolute minimum? (That's where changes in one part of the data require same-time changes in another part, because the data structure is not normalized.)

Regarding the latter point, in my experience it is best to design the data structure so that, if it must be non-normalized, that it be able to tolerate temporary inconsistency, which can later be resolved by some sort of periodic sweep. A major killer of performance is when notifications trigger further notifications, which trigger further, to an extent you never would have guessed beforehand. And often it's wasted effort because of self-canceling changes.

If you've done all this, you have a clean design. Then periodically as you develop it, profile. (Random-pausing is the method I rely on.) Then if you can see that performance would be improved by bringing in a more sophisticated algorithm, by all means do so.

Answer 3

No you should think about performance (especially in designing databases) from the start. There has been a lot of harm to our industry by people who think any optimization is premature optimization. The quote was orginally intended to prevent people from looking at micro-optimizations before a problem had occurred. It was not intended to not do any optimization at all. In a database for instance, there are many known techiniques which perform poorly. Avoiding those, in the design, is part of what you need to do. It is very difficult to refactor a database with 100,000,000 records because it was designed using poorly perforiming techniques and we can no longer avoid the issue by buying better hardware.

Answer 4

Worry about correctness¹ first, then maintainability, then safety and reliability, and then you can think about performance. Apply this order to each piece of code as you're developing it. It's possible that a performant solution may naturally fall out of simply keeping things clear and straightforward.

80% of performance is picking the right algorithm and data structure for the problem at hand; a poorly optimized quicksort is still going the beat the pants off of a highly optimized bubble sort in the average case (worst case it's a draw).

What everyone on SO is trying to smack down is the "which is faster, ++p or p++" mindset, where people get so caught up in outsmarting the compiler that they lose track of the larger problem, resulting in code that's brittle, bug-ridden, wrong, and best of all, not that much faster than a more straightforward solution would have been. I've dealt with that kind of code first-hand; one example was so brittle that we couldn't make any changes without breaking it completely.

---

¹ Where "correctness" means "fulfilling the specification", which is not synonymous with "bug-free".

Answer 5

You should start thinking about performance once you know what "good" performance is. In other words, it would be wrong to start thinking about performance before you've identified what the following thresholds are:

• Unacceptable performance--fix it now before it gets further out of hand
• Acceptable performance--it's time to focus on other features before you attempt to do any more with performance.
• Target performance--idealized performance numbers. I.e. if you have sufficient time and resources what would you need your system to do?

Once you've identified what those thresholds are, you have also identified the metric you are using to measure performance. That means you can set up some automated performance tests that you can run several times a day. That will tell you if you are getting better or worse.

To come up with those metrics, you need to understand what your system needs to do. For example, are absolute performance metrics called for (response within X time) or are throughput measurements called for (X responses per Y time)? Throughput and absolute time optimizations require different approaches, and if you don't know what is really important, you might be optimizing the wrong way.

Answer 6

You've probably heard that premature optimization is the root of all evil. The question is what makes it premature? In my opinion, it's never a bad idea to think about performance, but don't worry excessively until your code works. Once it works, do some heavy load testing, profile and identify bottlenecks, and do your performance optimisations.

That being said, there's nothing wrong with thinking about performance during the initial coding stage if you know certain techniques that will make a real difference. For example, choosing one storage structure from a library over another because past experience has taught you that one of them is faster/uses less RAM than the other. Or building in a simple (you can make it more sophisticated if later testing requires it) caching system for data that you know will be accessed a lot and would be much better cached. In this way, you're not fretting too much about performance (at least not initially) but you're using tips and trick you've learned along the way from other projects. Try to keep these simple so they're easy to include during initial development, and may offer some benefit as well.

Answer 7

Performance should be detailed in the system and user related specifications of your Requirements Document. I know many people sneer at the idea of undertaking Requirements Analysis in the development of an application, but amazingly such a document will concisely answer what and where you should devote your performance related resources to as the application nears completion. And it will answer that question in a timely manner

Requirements Documentation will save you hundreds of hours of time that would otherwise be wasted on non-essential processes.

Answer 8

A balanced approach would be better. Performance is important but not as important as getting things done, so:

1. first build a feature trying to think a little about what your doing and how your doing it (use a little time thinking about performance but not much)
2. Test it
3. once is running start thinking about if there is a real need to make it better (generally you wont, but in some case you might).

This is my common approach to performace vs functionality, and in general cases it all depends on what the program does and verifying if is there any need to make things work better and how much time would it cost me.

Lets think about a Q&A website like this one, i think the ones behind it surely thought a lot about how to make the Asking a question and getting the answer the most time/cost perfomant as possible. But, when thinking about notifications, it really does not matter much if notifications appear once in a while and tell you there is a new answer or something.

Answer 9

There is one way of a safe deferral of thinking about performance: using domain specific languages wherever possible.

If most of your development can be done with your own little DSLs, and they're designed well enough to express your problem domain in the most generic and high-level form, it is possible to get a working prototype first, without ever thinking about performance, and then only improve your DSLs implementations, not the actual problem domain code.

It is a much better approach from a maintainability point of veiw as well.

Answer 10

You should take performance into account. However, you must draw a line to mark the end of tuning, since (usually) your time is more important than the computer's.

A really good text article on performance is: The Computer Performance Shell Game.

The computer performance shell game, also known as "find the bottleneck", is always played between these four resources:

• CPU
• Disk
• Network
• Memory

At any given moment, your computer is waiting for some operation to complete on one of these resources. But which one: CPU, memory, disk, or network? If you're interested in performance, the absolute first thing you have to do is determine which of these bottlenecks is currently impeding performance -- and eliminate it.

Answer 11

The "best" way is a very loaded term, and the answer may be highly dependent on factors not known until runtime.

• Do you have plenty of memory? - you will get better performance with an "all-in-memory" data structure, but if you do not have enough memory swapping will kill your performance.
• Do you need persistance? A database gives integrity but is slower than the above "all-in-memory" data structure. MUCH slower.
• Can you cache results? Varnish can help with that! http://www.varnish-cache.org/

The list goes on and on.

What you can do, is to write "the simplest thing that could possibly work" from the knowledge you currently do have, and do it in a modular fashion so you can easily reorganize when you know more. Note that the "simplest" thing is not necessarily simple!

Answer 12

It's always something you should keep in mind. I think what most people are trying to say is that there is not much point in spending two days trying to optimize something that you don't even know is broken. Once you have a product up an running, and can do some usability testing, that should show you where you are having performance problems. Then, once you can identify the true performance issues, you can target the optimizations you need to perform.

Answer 13

In theory at least, you should start thinking about performance once you're in beta testing and not before.

This is not, however, license to make poor design decisions. For example, using an NVARCHAR string as a primary key is a sure path to poor performance; that said, it's a filthy habit regardless of the performance issues and you shouldn't use in the first place.

If your design follows conventional best practices (everything in 3rd normal form, proper information hiding in your classes, minimal use of singletons, etc), and there's a performance issue later on, it'll be easy to handle (create an index here, implement a cache there).

HTH

Answer 14

It depends. It is useful to keep in mind the 80/20 rule: most (let's say 80%) of the code in the application is never going to be executed often enough to make any noticeable difference in performance. You need to focus on the remaining 20% where the app is bound to spend about 80% of its execution time.

You may be able to identify some of the obvious performance hot spots in advance, such as if you know that a particular calculation is going to be iterated millions of times over. In such cases it is definitely worth thinking about optimizing it up front by choosing the right data structures and algorithms for the job.

However, that optimization is more a design activity. What is usually worthless is micro-optimizations, where someone spends an inordinate amount of time with clever bit-fiddling tricks in the name of "gaining performance". Especially if done without appropriate measurements before and after, such changes may not make any difference, or actually slow down the app under real life circumstances.

Answer 15

It depends on which stage of development you are in

1)If you in building functionality of your software keep it going and make sure it works fine(i.e,desired and efficiently).

2)Once building blocks integrated,you will get hint of resource hogger,there you have room for optimization.

Answer 16

If you have to start thinking about performance, you're in trouble. You should be thinking about performance all the time. In fact, I suspect good programmers are going to think about performance even when they didn't intend to, in a «men think about sex every seven seconds» fashion..

What is important is what actions you will take based on all that thinking. Thoughts are cheap, but actions can break code and blow deadlines.

Most of the time, the only sensible action will be to do nothing: you identified that your piece of code is not going to be called often enough for performance issues to be observable—maybe it's a piece of startup code that runs once per computer for 1% of your potential user base, maybe it's a small bit of redundant server code drowned in a sea of slow database accesses, maybe it's just an integer assignment in a non-critical section of code.

Quite often, you suspect that a given operation might cause a performance issue that could be solved by a simple change. There's for instance, the nagging feeling that running a complex SQL query on every request, or a asking for the same piece of data from a dictionary twice, is going to be bad for you. This is where knowledge of optimization techniques comes in handy, and perhaps the most surprising conclusion happens:

If you know of a quick technique that will almost certainly improve the performance of a piece of code, don't do it.

If you can think of it now, you can certainly do it in five minutes later on. Keeping it out of the code (but, perhaps, in a // TODO comment) leaves the code cleaner and saves you previous time to work on another feature, while wasting no time if you end up throwing that code away later on. If the original code does turn out to cause performance issues when tested, go back and apply your quick technique.

I'm not saying here that you should avoid writing code that's idiomatic just because it happens to be faster. Write idiomatic code according to best practices that improve productivity and readability and reduce bugs. It's just that if you have a choice between idiomatic by-the-book code and a faster but easily written alternative, always go for readability instead of speed.

The only difficult situation is when there seems to be no easy way to improve code performance, and yet it's painfully obvious that a piece of code is going to break as soon as it's delivered—a full database traversal on every click, a hundred SQL requests per page on the site, or anything similarly dreadful. This is where you actually need to stop and think some more. These are usually architecture issues that cannot be solved on a local scale anyway. Confirm your suspicions with a quick spike or prototype, look for similar experiences and common solutions, and consider either a change of architecture or a drop of features.

Answer 17

IMHO it is important to think about performance before you implementing the system, but only think about. You should analyze the application and find out what could possibly be potential performance bottlenecks.

Then implement the system as simple as possible. If performance problems arise, then optimize.

For example, say you have a GUI client that gets data through some kind of service (SOAP, REST HTTP, etc). Then the most important thing for high performance/scalability is to have as few calls as possible, having each call return a lot of data, rather that a lot of call returning a little information each, i.e. prefer chunky communication to chatty.

When implementing this kind of system, I wouldn't care that much about the number of calls between the system. But I would make sure that the code base would make it easy for me to refactor/optimize when the needs arises.

Answer 18

You should think about performance in very general ways from the start. You should select data structures and algorithms that will work well for your application and be reasonably efficient. Algorithms are fundamental to the software, and data structures more so. You will likely have to do major rewrites if you need to make major changes to either, while smaller details can be rewritten more easily.

You might also want to get efficient habits, but these will depend on language. In C++, for example, "++i;" as a stand-alone statement or expression is always at least as good as "i++;", and could potentially be much more efficient. In most cases, though, you should write clear code and trust the compiler. Getting into the habit of worrying about micro-efficiencies will almost certainly cause you more problems than it solves. For desktop apps, either the compiler is at least as smart as you are about things like i >> 1 vs. i / 2, or the best way to improve performance is to get a better compiler, so don't worry about that.

Beyond that, don't worry much until you've got something you can test. At that time, you can profile the program to see where the hot spots are, and probably get an idea as to whether you have a performance program or not. If you need to improve performance, find where the program is spending most of its time, and improve things there. If you've designed in adequate global efficiency, and written the program well, you're only changing a relatively small part of the program.

Answer 19

I think the best you can do is to follow good design practices (e.g., don't do things you know will hinder performance) until you get something working. If you can't measure improvement, you can't make improvement. Once you've got something you can test with, then it's often a good idea to take a profiling run and get an idea where the hot spots (if any) are. If something jumps out at you, you should consider refactoring or rewriting the problem area, but if it's not too bad (just because the code spends 90% of its time in two or three methods doesn't mean anything if it performs adequately overall) then just keep developing. Something I've seen more than once is developers who spend days optimizing the most complex part of the system, only to find out during integration testing that some unrelated part of their code is horribly slow.

Answer 20

When should I start thinking about it? How much effort should I put into it? That depends on the Cockburn Scale of the project. (In other words, what is the risk of not having good performance?)

Learn the fundamentals well in advance (see Robert Harvey's answer). In order to apply performance-oriented thinking during various stages of software development, the developer has to know it inside-out, so that the thought process will not be hindered by those extra considerations. (In other words, start thinking about performance before the project is conceived.)

During the early stage of development, make liberal use of performance profiling tools and keep track of the stats history. Pay special attention into organizing such information to make them useful for later decision-making.

Then, depending on your project's nature and its Cockburn Scale:

• Rapid Prototyping, or "bang out code like there is no tomorrow", or In-house development with low business impact: just keep the statistics. Don't think about performance yet. Implement the function in the easiest way. Stick with the first algorithm that comes to your mind.

  • At the later half of the project, there will be ad-hoc performance testing to identify "hotspots" based on actual use cases. If there is any performance problem that make the software unusable, it should be easily identifiable.
  • For in-house development with low business impact, just deploy the code and fix performance issues later.

• Desktop applications, which require a consistent, well-rounded approach to performance. It does not have to be highly optimized; however, there should be as few "hangs" (non-responsiveness) as possible.

  • Desktop applications usually have highly convoluted execution paths which complicates performance-oriented thinking. A layered design will allow the separation of Database/Network performance issues from GUI performance issues, to be handled by different experts in your team.
  • A history of Log traces will allow hotspots to be identified.

• High performance computing, which requires getting the most performance out of the hardware.

  • Appoint someone in your team to be in charge of analyzing and reporting the performance statistics.
  • Make theories about the performance characteristics, verify with experiments, and compare with your predictions from simplified Comp Sci models.
  *

Answer 21

At the beginning. Identify required performance characteristics. If you can't identify the target, you either need to step back to better understand your requirements or defer until you know your component requirements with the risk you might be rewriting. Then, test. Don't optimize, test. If you code fails the performance test, optimize. With a test framework in place, usage of existing performance monitoring tools should make the task reasonably easy.

Keep the performance tests in place for the lifetime of the project as a regression test. Maintenance code is notorious for triggering performance problems because the 'fixes' often have a very narrow focus.

Answer 22

I always rely on a simple formula:

1. Make it work
2. Make it right
3. Make it fast

... in that order.

According to c2, this formulation is attributed to Kent Beck.