3

I have developed a CUDA code that is a reimplementation of a (CPU only) C code.

I would like to benchmark the performance of these two programs. I am wondering how to do benchmarking well in this case, as the hardware they are run on are different.

For example, it would be unfair to run my CUDA C program on a $3000 GPU and compare those results with the C code running on a $100 CPU, and then celebrate what an amazing GPU hacker I am.

Are there any standard ways in comparing code written for GPU and CPU that takes into consideration the difference in hardware?

Unit computation divided by consumed energy, price?

2
  • 9
    it would be unfair to run my CUDA C program on a $3000 GPU and compare those results with the C code running on a $100 CPU -- That's kinda the whole point, isn't it? The only measurement that seems relevant here is the relative speed difference between your CUDA program and your C one on the same machine. If you want some degree of fairness, run both programs on a machine with a CPU and a GPU that a reasonable gamer would buy. Nov 30, 2016 at 21:07
  • 4
    I don't think this question has one clear answer, it all depends (thus possibly: too broad, opinion-based). In many real-life scenarios, what people are interested in is either end-to-end time to accomplish a particular task (e.g. time to produce a 24-hour weather forecast), or steady-state throughput (e.g. how many live video streams can be compressed simultaneously). In other circumstances, performance per energy expended (unit computation / Joule), throughput per power consumption (e.g. GFLOPS / watt), or a price-related metric, may be more appropriate.
    – njuffa
    Nov 30, 2016 at 21:10

1 Answer 1

4

You have to carefully define what you are optimizing for. I assume you are writing your programs to solve some problem. We can then ask:

  • How long does it take to solve the problem with C or CUDA? This cycle time includes both the development time and the running time. It may include the time required to acquire necessary skills and equipment.

  • How much does it cost to solve the problem with C or CUDA? This includes developer salary, energy costs, overhead, transaction costs, and opportunity cost.

So unfortunately, this is more about project management than about benchmarking.

There are a couple of extremes where the answer is simple.

There are use cases where the development effort dominates. This happens when your problems are very specific and each need substantial development effort, and each program is only run a couple of times. E.g. if I need 5 days to write the program, then it doesn't really matter if I get an answer in 12 hours or 45 minutes.

There are use cases where raw performance is most important, e.g. in high-performance realtime systems. Increased development, energy, and hardware costs are more tolerable than getting an answer 5 milliseconds too late.

And there are uses cases where the program is run so much that hardware and development costs are quickly amortized. Performance per watt starts to dominate the calculation.

I'm currently involved in a small research project where the run time dominates over development time, and the long cycle time results in significant opportunity cost. This means we are effectively time-limited, not cost-limited. To shorten the cycle, we're throwing more resources at development and hardware, whereas energy costs are negligible. If switching to a higher-performance technology doubles the development effort but doubles performance, that's absolutely worth it for us.

1
  • I'd add another bullet in your list: How much does it cost to not solve the problem ? I mean, if you solve the problem with C-CPU only, and it's TCO is cheap, but doesn't solve the time problem, how does it affect your user ? This answer may steer you into CUDA as being the single possible solution for your user. That being said, I agree with everything in this answer.
    – Machado
    Jan 2, 2017 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.