How do I avoid “Developer's Bad Optimization Intuition”?

Question

I saw on a article that put forth this statement:

Developers love to optimize code and with good reason. It is so satisfying and fun. But knowing when to optimize is far more important. Unfortunately, developers generally have horrible intuition about where the performance problems in an application will actually be.

How can a developer avoid this bad intuition? Are there good tools to find which parts of your code really need optimization (for Java)? Do you know of some articles, tips, or good reads on this subject?

Answer 1

• Use a good profiler to identify expensive methods.
• Document how long the hot spots actually took.
• Write a faster implementation of the hot spots
• Document how long the hot spots now take, hopefully not making them hotspots anymore.

Essentially you need to be able to prove to others where the problem was, and that this change made it go away.

Not being able to prove an improvement, qualifies - in my personal opinion - for immediate rollback to the original version.

Answer 2

The only way to know where to optimize is to profile your code. Instead of making changes that you think will provide a benefit, know for sure where the worst-performing code is and start there.

Java makes this pretty easy with the VisualVM tool, which has been bundled with recent releases of the Java Development Kit (JDK). The idea is to find out which methods are called the most and in which methods you are spending most of your time, both in your code and in external libraries. You can also get performance data on garbage collection so you can tune your collector and adjust min/max heap space required by your application.

Answer 3

As anyone here is talking about profilers I will focus on this part of the question.

How can a developer avoid this bad intuition?

You. do. not. Instead you never optimize early on.
Repeat it again and again and again, as it's a religious mantra.

You will find yourself doing that and will discover you should have not.
And then again.
And again.

Early Optimization is one of programmers' capital sins.

Tools and stuff are part of the later optimization which is an established craft.

Answer 4

These tools are called profilers. You can use them to actually measure which part(s) of your program take the most time to execute, so where to focus your tuning efforts.

It is equally important to measure again after the changes, to verify that your changes have the intended effect.

Answer 5

Look also at how much memory your program uses, not just its speed or runtime.

Lots of coders who work with garbage-collected languages such as Java are under the mistaken impression that garbage-collection prevents memory leaks. That is not the case. If you hold a reference to an object you don't need anymore, it won't get collected, and so it will leak.

I have seen Java web applications that were so leaky that they would run their server out of swap space!

If you use both a runtime profiler and some manner of memory profiler, you will learn to write faster and leaner code intuitively. This has the effect that your code is more likely to run fast on the first try.

Answer 6

my remedy is to start by getting clear answers to two questions:

1. how to measure performance (eg. measure data load time)
2. what is the target value (eg. data loads in 3 sec or less with 95% confidence)

Learned above trick from tiger team guys who were once invited to save a broken release of our product. That release was broken for performance reasons, it could make company loose strategic customer which justified tiger guys involvement (pretty expensive btw). I was assigned to assist them in clarifying project details; also used this as an opportunity to learn a bit or two about performance.

Answer 7

What I've found is the best antidote to premature optimization is this method.

Once you've used it to speed up some code (as in this example), it becomes an addiction of its own, and you understand the first principle of performance tuning isn't tweaking code, it's finding the problem.

Real optimization is to premature optimization as hunting to feed your family is to shooting tin cans. It's all about finding the quarry.

Answer 8

Old question, but I'll offer this answer which differs considerably from the others.

The big opportunities for performance gains come from parallel processing. Try to design your code to take advantage of multiple threads. (Even if, for simplicity, you don't do so in version 1). Little guesswork or intuition necessary.

Anything else is premature optimization, and requires your intuition, which is often wrong.

Answer 9

Your intuition can improve with time. I'd throw that out, maybe a bit controversial, but over many years of using VTune and CodeAnalyst and now CodeXL, I'd say I'm far more accurate in my intuitions than before about where the hotspots will be, at least to the point where I'm no longer caught completely off-guard when I profile some code. That doesn't mean I attempt to optimize things blindly.

Profiling has actually increased my dependence on profilers, not lessened it. I'm just saying I can more easily anticipate what the profiling results will be to some extent and moreover, successfully eliminate hotspots and improve the time it takes to complete the user-end operation without taking blind stabs in the dark and missing (something you can do even when using a profiler until you start understanding not only what the hotspots are, but why exactly they are hotspots with respect to, say, cache misses).

However, it wasn't until I started using profilers that I started improving that intuition. One of the reasons is because if you are well-familiar with your code, your hunches might be correct with respect to the biggest and most obvious hotspots, but not all the subtleties in between. Naturally if you have a user-end operation that takes an hour to complete and there's one gaping quadratic complexity algorithm processing an input spanning a hundred thousand elements, you can probably come out rich gambling your entire life savings on the idea that it's the quadratic complexity algorithm at fault here. But that doesn't give you any detailed insight or, say, let you know exactly what isn't contributing to the time.

There's so much value to be had when you start profiling and seeing where all the things you thought might have been a larger contributor of time wasn't contributing much time; not the gaping obvious sources of inefficiencies but the ones you suspected might have been slightly inefficient but, after profiling, realizing they barely contributed any time whatsoever. And that's potentially where you gain the most intuitive insight is finding yourself being shown wrong in all those subtle areas where it's not obvious exactly how much time is being spent.

Human intuition beyond obvious algorithmic complexity will often start out incorrect because what's efficient for the machine and what's efficient for the human mind are very different. It doesn't come so intuitively at first to think about memory hierarchies going from registers to CPU cache to DRAM to disk. It doesn't come intuitively to think that redundant arithmetic may be faster than doing more branching or memory accesses of a look-up table to skip some processing work. We tend to think in terms of how much work there is to do while discounting things like the cost of making decisions and memory loads and stores. What is efficient for the hardware is often very counter-intuitive in ways that will break all your human assumptions starting out, but naturally you need to be measuring to break your assumptions in ways that have you learning and aligning your intuition closer to the way the hardware actually works.

Where improving that intuition can help, through profiling, is interface design. Interface designs are very costly to change in hindsight, with costs rising in proportion to the number of places depending on that interface. When you start improving your intuition, you can start designing interfaces better the first time around in ways that leave breathing room for future optimization without costly design changes. Again though, that intuition is something you generally develop, and continue to develop indefinitely, by always having that profiler in hand.

Answer 10

Profilers help fix the bad intuition when it comes to code. Given how much hardware predicts these days it's not humanly practical to predict the performance of your code, but that was still even true in Knuth's time so many decades ago who advocated that profilers should be included as part of the standard tools for development to fix the "pennywise-and-pound foolish" nature of developers. But I'm going to go a very different route with this answer given how comprehensive the answers are in other regards and say user-end understanding is the other "fix".

I have witnessed, in my personal experience, a particularly brilliant developer (but with gaping blind spots about how users actually use the software) optimizing a subdivision algorithm with profiler in hand (a very good and expensive and comprehensive one: Intel's VTune with call graph sampling on top of GPU profilers) for meshes achieve amazing results with billions of facets on the GPU when subdividing simple primitives like cubes with 6 cage/input polygons. Except he tuned and tuned it against that test case which was unlike any real-world use case (users don't want a billion facets to a subdivided cube starting to resemble a perfect sphere, their subdivision inputs tend to be things like characters and vehicles and other complex inputs). Meanwhile when he was given a production model of a car spanning 120k input polygons to subdivide to only a relatively small number of facets, the brilliant algorithm he benchmarked and profiled repeatedly came to a total crawl taking 15+ seconds to initialize, rendering his solution virtually unusable for real-world use cases.

Funnily enough I got further with a brain half as functional as him and no PhD to my credit in my career for the mere reason that I understood what users wanted, what marketing wanted, what designers wanted. I can't really emphasize enough how useful it is to be able to put yourself in the mindset and shoes of the user and look at your software and what it needs to do as your actual users while striving to divorce yourself from the efforts you put into building what you built and looking at it with a fresh pair of eyes. I even encountered from the developer above that this is an impossible thing to do; he thought I was guilty of having a similar sort of ego that all technically-savvy but user-oblivious developers have, and I've constantly proven him wrong when users and designers flock to me to talk about exactly what to do. That sounds very egotistical but I'll balance that with the disclaimer that I'm not such a brilliant coder, but I do understand what users and designers want, and that made me particularly favored in my field where this seemed to be a particularly rare quality for some reason. As programmers we're probably more used to acing tests than understanding and socializing with ordinary, non-technical people.

So there's profiling and properly measuring but there's also the fundamental need to actually make sure you're measuring an operation with the type of inputs that real-world users are actually going to provide to the application. Otherwise you can even have VTune or CodeAnalyst or gprof or any other profiler in hand and still, while trying to optimize hotspots against what might seem like a normal test case to the developer but an obscure one to users, end up pessimizing the common use case in favor of some obscure use case that few users, if any, ever consider applying.

At the end of the day all impracticalities we tend to carry as developers can be balanced against the iron hammer of what makes users genuinely happy enough without solving world hunger, and the practical need to get money so that we can pay rent or buy beer or look at naked ladies or whatever you want/need to do. Everything else is is potentially working against that fundamental business need, and any developer so noble, so heroic, so as to forget that this is about making money and ultimately satisfying users to get them to pay up might do well to bring himself down to earth and turn off god mode creating virtual worlds in favor of the real-world need to simply ship and get some money for food. We can get lost in software metrics and practices but fundamentally it's really about whether the users can pay you money to support your drinking habits.