1

I've been researching micro-benchmarks (e.g. using JMH or Caliper, though not necessarily limited to Java). Now, there are many articles which describe how to write a benchmark but I haven't found much on when to write a benchmark (and when not to). There were a few things that go in that direction, e.g.

Continuous delivery, continuous testing… is one thing, but continuous measuring is another step.

or

Always Measure

Before you start optimizing, make sure you have a problem that you need to solve. Make sure you can accurately measure your existing performance, or you won't be able to measure the benefit of the alternatives you try.

But that's only of limited value to me.

So my question is: In which scenarios do micro-benchmarks make sense? When don't they? What should and shouldn't they be used for? And does anyone here have real life experience with micro-benchmarking in a large project?

11
  • 2
    It's not clear to me what, if anything, your bullets have to do with micro-benchmarking, specifically. Running a profiler to find the hot spots is always an option. Jun 24 '15 at 19:44
  • @RobertHarvey I guess all of the measure bullets would be as valid for a macro-benchmark as for a micro-benchmark, though on a different scale of course. The don't measure bullets are a bit more specific to micro-benchmarks since macro-benchmarking a network application can make sense if the outside dependencies have an average time much lower than the total time you're measuring. And the time limit is of course very micro-benchmark specific. Jun 24 '15 at 19:49
  • @RobertHarvey Mostly agree. ... But, always an option?
    – svidgen
    Jun 24 '15 at 19:50
  • @RobertHarvey Oh, and running a profiler is great to find hot spots of course. I'd see micro-benchmarks as a tool to compare various ways of fixing those hot spots. Not as replacements for a profiler. Jun 24 '15 at 19:53
  • 1
    This bullet point, "Don't measure ... if the operation will take longer than a few milliseconds to complete," appears contradictory to the bullet point right above it, "if the measured piece of code won't be called very often (or if you're unsure) unless your application is REALLY time sensitive". Any clarifications?
    – rwong
    Jun 24 '15 at 19:58
4

Micro-Benchmarks can be used in addition to a profiler to find and fix problematic parts of your code. So, you can run a profiler to identify the problematic regions and then test those specific regions with micro-benchmarks.

Generally speaking, micro-benchmark frameworks (such as Caliper and JMH) are designed to prevent many of the common mistakes and make it easier to create meaningful benchmarks (as is described in the answer to this question). These are in many cases problems which don't occur in the same form when you're testing larger pieces of code; therefore it makes sense to keep micro-benchmarks small. (That's why they're called micro-benchmarks after all.)

Some things I've come up with are the following.

Measure:

  • if you've found a bottleneck and you may have a better solution
  • if you're comparing several technologies which can do the same job (e.g. xml parsers) and want to know which one is fastest before using them in your real code
  • if you have nothing else to improve about your code

Don't measure:

  • if the code has time wise unreliable outside dependencies (e.g. it talks over a network)
  • if the measured piece of code won't be called very often (or if you're unsure) unless your application is REALLY time sensitive. (= If you rarely run a piece of code, don't try optimize it. It probably won't be worth the time.)
  • if the operation will take longer than a few milliseconds to complete
  • just because your boss, manager or someone else who is in charge thinks that it's a good idea to do so

Also, while some people run micro-benchmarks regularly (as you would run unit tests), since I use them only to analyze problems I already know about (thanks to the profiler), these benchmarks are not something I'll need to run very often. There may be situations in which this would make sense (and if you know of one, please comment!) but that would probably be a totally different way of using such tools.

Not the answer you're looking for? Browse other questions tagged or ask your own question.