12

My employers runs a monthly unit testing day competition. One entire day is dedicated to writing unit tests -- obviously we do more testing throughout the month, but this is an entire day -- and the "winner" of the competition is given a prize. However, we are finding it's hard to determine who the winner is.

We were assigning points for each test case. So if you wrote a unit test like this...

for (int i = 0; i < 100; i++) {
  assertTrue(i*i, square(i));
}

you would be given 100 points. Obviously this is a simplistic example but it demonstrates the problems with assigning "points" to each test case.

We're primarily a Java & Javascript shop. So I suggested counting number of code branches tested as a metric. We can easily count the branches tested via a code coverage tool (such as EclEmma). However, not sure how we would do this with our Selenium tests and getting a code coverage on the Javascript sources (any ideas?)

Does anyone have any suggestions on how we could better determine the winner of this competition?

Edit

I know how to write unit tests, I know how to write effective unit tests, I don't need help determining what to test. I have no control over this competition -- the competition will go on. So either I add some input to make it better or carry on gaming the tests (yeah, I game them. Of course I game them. There are prizes to be won)

Edit

This question here is obviously not a duplicate, though it contains useful information about how to find good test cases, it does not provide any useful metrics to evaluate the competition.

  • Not quite. I realized it from the start – Shaun Jun 2 '15 at 20:27
  • 2
    You don't seem to quite realize the full extent yet. Any measure of who wrote the best test cases will either be completely subjective or have these problems to some degree. What metric works best will depend on your goals for this competitions and on how mature (i.e., unlikely to exploit the scoring rather than writing the best tests they can) the contestants are. – user7043 Jun 2 '15 at 20:30
  • Again, no. I realized they can be gamed. I have no control over this competition but was asked "how can we do this better" – Shaun Jun 2 '15 at 20:32
  • 13
    Would it be considered an improvement to not make it a competition? Why does everything have to be a competition? Why can't you collaborate? Maybe getting rid some of your more pointless unit tests and building a nice suite of useful smoke and regression tests would be helpful. – Thomas Owens Jun 2 '15 at 20:36
  • 1
    I'm with Thomas... the winner should be the code base/customer because the code quality improved. Set an overall/group goal based on code coverage of the unit tests... +5% over current or whatever. ... and don't game the system for prizes... whatever happened to a job well done is its own reward? – JeffC Sep 20 '15 at 4:25
15

Does anyone have any suggestions on how we could better determine the winner of this competition?

The only thing which makes sense to me is by voting - every dev can assign some points to every other dev's test (except his own). Maybe 3 points for the test he thinks it is the "most effective" one, 2 points for the second and one to the third. The test with the most points wins. It may give better results when the point assignment is done without knowing beforehand who wrote the particular test.

As a bonus, you will get all your tests peer reviewed.

  • 2
    This was my thought also. There's no other way to measure the value of tests. – Eric King Jun 2 '15 at 20:55
  • 2
    Yes, "good testing" is such a subjective thing it needs considered judging, either by peers or respected authorities. Chasing metrics will just lead to a lot of wasted effort and little real value. Might be interesting to have multiple prizes: most imaginative test, a "testing something previously considered untestable" award, best performance test, most effective test, most obscure test, cleverest test, most valuable test, test most likely to be appreciated by end users... – timday Jun 2 '15 at 23:18
6

So if you wrote a unit test like this...

for (int i = 0; i < 100; i++) {
 assertTrue(i*i, square(i));
}

you would be given 100 points.

I would give this person 0 points (even if the test were testing something actually relevant), because assertions within a loop make little sense and tests with multiple asserts (especially in a form of a loop or a map) are difficult to work with.

The problem is essentially to have a metric which cannot [easily] be cheated. A metric which is exclusively based on the number of asserts is exactly the same as paying developers per LOC written. As with pay-by-LOC which leads to huge and impossible to maintain code, your actual company policy leads to useless and possibly badly written tests.

If the number of asserts is irrelevant, the number of tests is irrelevant as well. This is also the case for many metrics (including combined ones) one could imagine for this sort of situations.

Ideally, you would be applying systemic approach. In practice, this can hardly work in most software development companies. So I can suggest a few other things:

  1. Using pair reviews for tests and have something similar to the number of WTFs per minute metric.

  2. Measure the impact of those tests over time on the number of bugs. This has several benefits:

    • Seems fair,
    • Can actually be measured if you collect enough data about bug reports and their fate,
    • Is actually worth it!
  3. Use branch coverage, but combine it with other metrics (as well as a review). Branch coverage has its benefits, but testing CRUD code just to get a better grade is not the best way to spend developers' time.

  4. Decide all together what are the metrics you want to enforce for the moment (such decisions may not be welcomed or even be possible in some companies and teams). Review and change the metrics often, picking the ones which become more relevant, and make sure everyone clearly understands what is measured and how.

  • 1
    +1 for zero points. Other objections would be AAA - Arrange, Act, Assert; Parameterized Tests; No Copying of code of the implementation... – thepacker Jun 2 '15 at 21:05
5

I suppose your employer organizes this unit testing day in order to give you people incentives to find bugs, to achieve greater code coverage, and also to end up having more tests, which are useful forever after.

So, I would think it would make sense that the winner should be the developer who finds the most bugs, or the developer whose tests achieve the greatest increase in code coverage.

A test would earn you a point if it causes a new entry to be opened in your issue / bug / defect tracking system. If an entry is already open for that issue, it does not count. Also, as suggested in the comments, bugs in your own code do not count; only bugs in other people's code should count. Unfortunately, this approach does not deliver instant gratification because it may take a few days until all the failed tests are sifted through and the corresponding issues are opened. Also, this may not always work; as your system matures, it may start becoming exceedingly rare to discover bugs by adding tests.

The increase in code coverage might provide a more objective measurement of the improvement represented by the new tests. First, the total code coverage will have to be recorded the day prior to the competition. Then, each developer will need to somehow show the increase in code coverage which results from their tests alone, without taking into account the increase in code coverage resulting from tests written by other developers. This means that you will probably need a referee who will go to each developer's machine and record the new code coverage before anyone's tests have been committed.

Incidentally, taking code coverage into consideration provides fair reward to people who write real tests, instead of doing silly things like the example that you provided in the question.

  • 2
    Sounds promising... but then "gaming the system" behaviour turns into lining yourself up a nice collection of known-only-to-you bugs to be "discovered" next testing competition... dilbert.com/strip/1995-11-13 – timday Jun 2 '15 at 23:02
  • 3
    One option is to only award points for bugs in code that someone else wrote. – Cel Skeggs Jun 2 '15 at 23:22
  • 2
    Sounds like this question and this linked article from that question... – h.j.k. Jun 3 '15 at 0:52
  • @col6y you are right, that's quite important too. Unfortunately, there are still ways to rig the system though. For example, if your code invokes my code to get its job done, my code could see to it that your code suffers an "accident". – Mike Nakis Jun 3 '15 at 6:51
  • 3
    I disagree. Unit tests, when they are newly written, are not for finding bugs in the first place, that's a fallacy. They can find regressions weeks or months after they were written, but that's most probably too late for providing a useful metrics for the competition. You typically write a unit test after a specific bug has occured to make sure you won't get the same type of bug later in the future. – Doc Brown Jun 3 '15 at 8:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.