I just developed an algorithm and additional to the usual unit tests I wrote a profiling "test" that I was using to measure and optimize its performance. It is structured similar to a test (arrange: setting up a sizable chunk of data to process, act: run the algorithm) but without the assert stage (it doesn't test anything).

Now that the algorithm is sufficiently optimized I'd like to keep the profiling code for future reference but since it takes several seconds to run without actually testing anything I don't want it to run every time I run my unit tests.

Of course I could turn it into a test by asserting the running time is below a certain threshold but that feels artificial.

  • Should I turn it into a proper test by asserting a certain running time?
  • Should I check it in as is?
  • Should I check it in as is but disable it so it won't run automatically?
  • Should it be in a separate file/assembly or together with the unit tests?
  • Should I do something else?

I hope there are some best practices so this won't be opinion based.

  • 2
    I can't answer regarding best practice, but I've been faced with a similar issue: I turned it into a 'test' as you describe with a stopwatch and a certain threshold which I assert against. However, I've assigned a category to such tests so that I can can omit them from the build process but run them (manually) whenever I need to. However, if it's essential that the threshold is never exceeded then I would consider adding them to the main integration test suite so that they are ran as part of a formal build process and nothing leaves the studio until you're happy that the performance is right. – pb01 Feb 4 '16 at 10:02
  • 1
    add it to your integration test suite and run it with those. You should already have a bunch of end-to-end (or similar) performance tests anyway. – gbjbaanb Feb 4 '16 at 10:22
  • @gbjbaanb: I don't. So far I only have unit tests. Where should integration tests go then file/assembly wise? – Roman Reiner Feb 4 '16 at 10:24
  • 2
    @RomanReiner integration tests, as the name says, are ones that you run on the final built system, so you can see if the joins between units work. Its also a place to see if data flow works and performance is acceptable. So it doesn't go with any file or assembly, but the overall product - its an 'external' thing. – gbjbaanb Feb 4 '16 at 10:26
  • @RomanReiner: Optimizing and Measuring are two different things. The way to optimize is to locate and remove processing that you can avoid. You can do this by single-stepping or sampling, to see why most time is spent. Then you can measure to see how much time was saved. Measuring does not tell you what to optimize, any more than a bank balance tells you what you should not have spent. – Mike Dunlavey May 25 '16 at 12:15

SQLite separates the test files as follows:

The "typical" workload is generated by the speedtest1.c program in the canonical SQLite source tree. This program strives to exercise the SQLite library in a way that is typical of real-world applications. Of course, every application is different, and so no test program can exactly mirror the behavior of all applications.

The speedtest1.c program is updated from time to time as the SQLite developers' understanding of what constitutes "typical" usage evolves.

The speed-check.sh shell script, also in the canonical source tree, is used to run the speedtest1.c program. To replicate the performance measurements, collect the following files into a single directory:

the "speed-check.sh" script, the "speedtest1.c" test program, and the SQLite amalgamation source files "sqlite3.c" and "sqlite3.h" Then run "sh speed-check.sh trunk".

Cachegrind is used to measure performance because it gives answers that are repeatable to 7 or more significant digits. In comparison, actual (wall-clock) run times are scarcely repeatable beyond one significant digit.

The Firefox browser separates the performance tests from the unit tests, and runs the performance tests automatically:

Every build of Firefox has Telemetry enabled. Local developer builds with no custom build flags will record all Telemetry data, but not send it out.

and includes the following features:

The about:telemetry page allows to view any data you submitted to Telemetry in the last 60 days, whether it is in existing pings or in new custom pings. You can choose which pings to display on the top-left.

You can find all important Telemetry resources listed on telemetry.mozilla.org.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.