0

I was coding some functions in C++ and wondered how different versions of those functions would affect generated assembly code. I put different versions into the Godbolt Compiler Explorer Tool and looked at the generated assembly. It was an interesting experience to see those differences and how some versions that seemed to be more efficient take up much more assembly lines than more verbose ones and in contrast some more low level versions take up more than some "mid high level" versions (contrary to my expectations).

As one cannot judge the performance of those outputs by just looking at the line count I wondered how one can roughly estimate the performance difference between different versions?

How can I analyse different outputs to see more easily if some code output contains more potential expensive ASM calls than another output or do I have to learn assembly first in order to do this?

Improve this question
8
  • 4
    The instruction set manual for a processor will typically list how many cycles each instruction will take, which can serve as a first approximation of performance in absence of cache misses, branch mispredictions, or pipeline stalls. You can then count how many cycles a piece of code will take. But this is really tricky. Some instructions may have a few cycles of latency until the result becomes available, that means you can execute other instructions for “free” in the meanwhile.
    – amon
    Commented Sep 10, 2017 at 21:27
  • 4
    Or, you just measure how long it takes to execute, the same way you would any other piece of code. But yeah, if you're doing some serious micro-optimization, the first place I'd look is in the microprocessor's instruction set specifications.
    – Robert Harvey
    Commented Sep 10, 2017 at 21:28
  • 2
    It is called micro-benchmarking. Execute the said code in a loop for billions (note: millions aren't enough) of times. This requires lots of computer architecture knowledge to do it right, because it's full of pitfalls, and it takes expert preparation to minimize interference and inaccuracies from many sources. Agner Fog has a website that talks about some of these things. At the minimum, the code needs to be tested on each vendor (e.g. Intel, AMD) and each family of CPUs (e.g. Ryzen, Haswell, Skylake, etc.) Benchmark results from one vendor/CPU generation is not valid for another.
    – rwong
    Commented Sep 10, 2017 at 22:00
  • 5
    // or do I have to learn assembly first in order to do this? // Definitely. As soon as you rearrange the code by e.g. calling it from a micro-benchmarking loop, or using the code in a real-world application, the compiler isn't guaranteed to emit the same piece of assembly code anymore. So if you are interested in performance of machine code, you have to know assembly language as well as compilers. You might not need to know how to write either, but you will need some intuitive understanding of what they do.
    – rwong
    Commented Sep 10, 2017 at 22:15
  • 5
    @amon: The instruction set manual for the 68020 processor (primitive compared to current designs) says paraphrased: "It is very difficult to calculate the exact execution time of some code, even if you have a deep understanding of the actual implementation of the processor".
    – gnasher729
    Commented Sep 10, 2017 at 22:25

5 Answers 5

Reset to default
8

As others have pointed out, just counting clock cycles of assembler instructions will not give you a decent result for most modern CPU architectures. The execution time of a fixed piece of machine code can vary on different CPU platforms, even if the code is exactly the same. So the only reliable way of comparing the performance of such code snippets is

  • get the compilers you want to compare
  • get one or more hardware & OS platforms for the comparisons
  • embed the code snippets you want to compare into some benchmark program (in its simplest form: just a loop which calls the snippet several million times and measures the total running time)
  • compile the benchmark with each of the compilers, using the available optimization options (maybe tailored for each of the platforms you want to compare)
  • then run the programs and measure on each platform

Obviously, learning assembly is not mandatory for this process (but it will probably be required if you want to understand the root cause for the performance differences you will observe).

That is probably not the answer you wanted to hear, I guess you were expecting a reference to some kind of web service like Mr Godbolt's site which provides you with the tools and hardware, so you don't have to buy and install them by yourself. However, I don't think there is a free service available: sensible performance comparisons need real, expensive hardware, not some cheap virtual machine replacement, and any company which can do this for you will probably try to get a decent return on their investment.

Improve this answer
4

There are two flaws in your process:

  • You are testing on a specific tool (Godbolt Compiler Explorer Tool) and looking at its output, which will surely differ from the output of a production-quality compiler (and that also highly depends on the optimization settings you choose).
  • With modern CPU's, it's virtually impossible to predict the run time from just reading the assembly code.

And finally, the general advice on performance optimization: do it only when you found it's necessary.

  • Start with writing well-structured, readable code.
  • Decide whether there's any performance problem. And "if it ain't broken, don't fix it".
  • If there's a problem, use a profiler to find out the real bottleneck (I bet, you'll be surprised where it is).
  • Find a solution for that bottleneck (maybe by micro-optimization, maybe by changing your algorithm...)
  • Profile again, to check if your change really worked.
  • If still too slow, repeat with the next bottleneck.
Improve this answer
1
  • 11
    Godbolt uses the production-quality-compilers to generate its output. It is a compiler comparison website.
    – Caleth
    Commented Sep 11, 2017 at 8:29
2

First of all: performance depends on the optimization that the compiler would make on your code, so you may want to test on the compiler you will use to build the final program.

Moreover, you will need to test your code for every major producer and chipset because the assembly language is different and similar instructions can be handled quite differently.

Improve this answer
2

I was coding some functions in C++ and wondered how different versions of those functions would affect generated assembly code.

And now you know its difficult. Was this just idle curiosity and do you /really/ need to know how fast it is.

I've had people to ask me which is quicker

x = x + 1; x += 1; or x++;

Any decent compiler will probably generate the same code. And your program's got a serious problem if incrementing a variable is a performance issue...

Improve this answer
2
  • No it was just personal interest, nothing to do with micro optimizations. Sometimes it is quite interesting what code compilers generate.
    – TorbenJ
    Commented Sep 11, 2017 at 16:06
  • Well, for custom types, there might be significant differences. For basic types, there shouldn't be any though.
    – Deduplicator
    Commented Jun 29, 2022 at 0:54
0

While I appreciate the fact godbolt can't account for cache misses, branch mispredictions, pipeline stalls, CPU load, etc, wouldn't the cycle count be an interesting metric for performance estimation?

A bit like the big-O notation: only theoretical, but still useful.

Improve this answer
3
  • For lots of code, cache misses take the biggest amount of time by far. If you look at the various M1 processors, they run fast because they have massive L1, L2, L3 and an additional shared memory cache.
    – gnasher729
    Commented Jun 29, 2022 at 12:36
  • I understand what your mean and I agree. That said, I work with machines without cache. Also, when evaluating small pieces of code, do you think that the metric wouldn't be useful?
    – gberth
    Commented Jun 29, 2022 at 12:40
  • Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Commented Jul 5, 2022 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.