So I'm trying to push for more automated testing in my company, which focus a lot on prototype and proof of concept systems.

We currently use Google Test for unit testing. This tests specific test cases for correctness, but a lot of the stuff we do the metrics for how the code is performing follow more of a "does it work better or worse than before, and if it works worse, is it an acceptable loss in performance" task than a boolean "yes no". These tests I call "Characterization" for lack of a better term (I've got a EE background, so sue me): it describes how the code behaves. We NEED to have the metric for all builds, and we NEED to be able to compare it between builds in order to see what direction we are heading. We can often make a general cuttoff to make a high-level pass fail, but these tests can be extensive: they take a long time to run and often require number crunching and plotting stuff. This seems to be out of scope for a Unit Test, more for a Functional Test in my mind (but I'm no test expert). How do people/companies handle this type of test? Is there any framework for this type of test out there?

  • You want to look at a system like Jenkins. Jenkins runs large scale long term tests. – Bill Door Mar 27 '14 at 22:24

You are talking about performance, which is essentially an Non Functional Requirement.

My view is that most of these are for the whole system.

A user is never going to say "I want the WidgetListCollection class to return a widget within 20 nanoseconds" -- he might say "I want a list of all products on my screen within 1.5 seconds".

The only way to really test these is to test the complete system. If the users screen is refreshed within .25 of a second exceeding the NFR by 600% than it does not really matter if the WdigetListCollection class is sub-optimal. Any effort spent optimizing it could be best spent elsewhere doing something that matters to the user.

Worse if you test this stuff at unit test level not only does it encourage you to waste time on micro-optimizations, your test may raise false alarms:

Say I may make a change to the class so it is now 5 times slower, however, it now also caches the widget usage history thus making the entire system run twice as fast. But I get nagged at the weekly meeting for poor performance anyway.

Unit testing is good. But it has its limits, in the end its the system as a whole that must work to spec. Be aware that the system can still fail miserably even though all the individual components are excellent.


How do people/companies handle this type of test?

There are many kind of testing, but your interests are unit and functional :

  • unit tests - needs to be very fast, in order to be execute on every change in the code. Their purpose is to test modules.
  • functional tests - they are similar to unit tests, except they do not need to be fast. They are executed when working on them, when there are no changes in the trunk for few hours, or when doing nightly builds.

Is there any framework for this type of test out there?

To do nightly builds, and to execute them when there are no activity in the trunk for few hours,, people are using frameworks like jenkins.

Not the answer you're looking for? Browse other questions tagged or ask your own question.