I just developed an algorithm and additional to the usual unit tests I wrote a profiling "test" that I was using to measure and optimize its performance. It is structured similar to a test (arrange: setting up a sizable chunk of data to process, act: run the algorithm) but without the assert stage (it doesn't test anything).

Now that the algorithm is sufficiently optimized I'd like to keep the profiling code for future reference but since it takes several seconds to run without actually testing anything I don't want it to run every time I run my unit tests.

Of course I could turn it into a test by asserting the running time is below a certain threshold but that feels artificial.

  • Should I turn it into a proper test by asserting a certain running time?
  • Should I check it in as is?
  • Should I check it in as is but disable it so it won't run automatically?
  • Should it be in a separate file/assembly or together with the unit tests?
  • Should I do something else?

I hope there are some best practices so this won't be opinion based.

  • 2
    I can't answer regarding best practice, but I've been faced with a similar issue: I turned it into a 'test' as you describe with a stopwatch and a certain threshold which I assert against. However, I've assigned a category to such tests so that I can can omit them from the build process but run them (manually) whenever I need to. However, if it's essential that the threshold is never exceeded then I would consider adding them to the main integration test suite so that they are ran as part of a formal build process and nothing leaves the studio until you're happy that the performance is right.
    – pb01
    Feb 4, 2016 at 10:02
  • 1
    add it to your integration test suite and run it with those. You should already have a bunch of end-to-end (or similar) performance tests anyway.
    – gbjbaanb
    Feb 4, 2016 at 10:22
  • @gbjbaanb: I don't. So far I only have unit tests. Where should integration tests go then file/assembly wise? Feb 4, 2016 at 10:24
  • 2
    @RomanReiner integration tests, as the name says, are ones that you run on the final built system, so you can see if the joins between units work. Its also a place to see if data flow works and performance is acceptable. So it doesn't go with any file or assembly, but the overall product - its an 'external' thing.
    – gbjbaanb
    Feb 4, 2016 at 10:26
  • @RomanReiner: Optimizing and Measuring are two different things. The way to optimize is to locate and remove processing that you can avoid. You can do this by single-stepping or sampling, to see why most time is spent. Then you can measure to see how much time was saved. Measuring does not tell you what to optimize, any more than a bank balance tells you what you should not have spent. May 25, 2016 at 12:15

3 Answers 3


SQLite separates the test files as follows:

The "typical" workload is generated by the speedtest1.c program in the canonical SQLite source tree. This program strives to exercise the SQLite library in a way that is typical of real-world applications. Of course, every application is different, and so no test program can exactly mirror the behavior of all applications.

The speedtest1.c program is updated from time to time as the SQLite developers' understanding of what constitutes "typical" usage evolves.

The speed-check.sh shell script, also in the canonical source tree, is used to run the speedtest1.c program. To replicate the performance measurements, collect the following files into a single directory:

the "speed-check.sh" script, the "speedtest1.c" test program, and the SQLite amalgamation source files "sqlite3.c" and "sqlite3.h" Then run "sh speed-check.sh trunk".

Cachegrind is used to measure performance because it gives answers that are repeatable to 7 or more significant digits. In comparison, actual (wall-clock) run times are scarcely repeatable beyond one significant digit.

The Firefox browser separates the performance tests from the unit tests, and runs the performance tests automatically:

Every build of Firefox has Telemetry enabled. Local developer builds with no custom build flags will record all Telemetry data, but not send it out.

and includes the following features:

The about:telemetry page allows to view any data you submitted to Telemetry in the last 60 days, whether it is in existing pings or in new custom pings. You can choose which pings to display on the top-left.

You can find all important Telemetry resources listed on telemetry.mozilla.org.



Well, let's put

I'd like to keep the profiling code for future reference


I don't want it to run every time I run my unit tests.

together. That immediately implies:

Should I check it in as is but disable it so it won't run automatically?

Yes. Do that. Also make sure to have a "build profiling version" or "build profiling binary" option for enabling it. Remember that others might want to profile it too!


The code for profiling should definitely be checked in for the same reasons that unit tests should be checked in! Other people must be able to run the benchmarks on their machine and compare their results with your results. Maybe the profiling results differ depending on the platform?

I would not add a test assertion to check the performance of an algorithm. The runtime depends on many external parameters (CPU power of the machine, other processes, etc), so a failing profiling test is not that meaningful.

I made good experiences with keeping the unit tests and the benchmarks in two separate projects/executables. Both are built by the CI pipeline, but the benchmarks only run on demand. This makes them easy to maintain and the CI pipeline at least ensures that they can be built.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.