I'm currently testing Visual C++ 10 on some trivial pieces of code. For example, like this one (taken from here):

int main()
    int i;
        printf("%d ",i);
    return 0;

For the code above the compiler emits machine code that in effect does this:

printf("%d ",2);
return 0;

and this makes me happy. Sometimes the compiler emits really dumb code and then I file a Microsoft Connect feedback item.

I have the following concern. Is testing a compiler on trivial code worth it?

On one hand, we want real code to be compiled as good as possible, not some stupid sample snippets.

On the other hand, compiler optimization is recursive in sense that if code B follows code A and the compiler can see that code B does nothing it can eliminate code B and eliminate whatever connected A and B logically and thus optimize (and maybe eliminate) code A better. And if the compiler fails to optimize away code B it will likely not be able to optimize code A either. So every improvement of generated code matters - better code at some point can mean better code elsewhere and this is beneficial.

Is it worth testing compilers on trivial samples or are only tests on "real code" worthy?

  • would you be even happier if he emitted putchar('2'), putchar(' ') instead of the printf? – Ingo Sep 14 '11 at 15:13

The optimizations you need a compiler to do are for the kinds of code that take time in practice. So it's a waste of time to try to find little weird cases that it does or doesn't do.

In fact, in some kinds of applications, compiler optimizations make no difference whatever, such as nearly every Windows app. If you take stack samples of such an app, as I do regularly, you find at nearly any random instant of time the call stack is at least several layers deep. The tip of the stack is either in some blocked system call, or executing instructions somewhere in the sub-basement of a system library.

In other words, in this type of app, it's almost impossible to catch the IP in code that your compiler compiled, and that's the only time the cycle-squeezing could even matter.

ADDED: Don't get me wrong. These apps can certainly be optimized. It's just that you can't expect the compiler to do it. This is the method I use.

| improve this answer | |

Is it worth testing compilers on trivial samples


or are only tests on "real code" worthy?

Barely. This is mostly a waste of your time, also.

Before wasting your time on compiler optimization, you have to answer three questions.

  1. Does it actually work?

  2. Is it the right algorithm and data structure?

  3. Is there an actual performance problem that you need to actually solve?

"we want real code to be compiled as good as possible"

This is not a good goal. You must have a specific performance guideline and you must stop wasting your time optimizing when you meet that goal. "as good as possible" is too vague.

| improve this answer | |

I think this is most likely a waste of time. Worse, it may divert resources away from finding issues that are likely to affect real code. If this happens to hit on a problem that actually matters, it will only be by luck. You're much more likely to hit the target if you aim at it.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.