Verification for value of test standards

Question

We are currently developing a concept for our tests standards. Up until now, we don't have standards. All we do is to tell the developer to write tests. Now we have the following basic idea:

We will use the coverage metric described in Developer Testing blog article Selecting Developer Testing Metrics to visualize the status quo.

We won't make demands how much percent the code has to be covered. Nonetheless we want some kind of control. Therefore we will check if the coverage has significantly reduced over time. If this is the case the whole team will be informed. There should be no personal blaming.

To encourage the developer to write more tests we will ask them to do the estimate of a feature with the tests in mind, so that they have enough time do write the tests.

We don't have much experience in test standards. Is this a good approach? Is this metric useful? And what do you think about our strategy to control the team members?

EDIT

Some additional information regarding Sklivvz's post. We have nearly no code covered. Overall it is about 25%. Some components are just not covered. Our problem is, that our developers - mostly people with 10+ years experience - don't write tests at all. Therefor we invests much time in fixing bugs found by our testing team. What we want, is to have more assurance that we don't break something by implementing another thing and that we don't make the same bugs.

Additional it would be nice to let the tests guide our team members to make a better architecture in their components. I personally have seen it in a component where I came in later and had refactored the whole component at some point, because it was not even a little bit testable.

We develop a software platform which grows several years now. Therefor we have much code which is not covered by tests. Like I said above we want to motivate our team members to write tests for their components which they are currently developing.

Answer 1

Caveat emptor: why are you implementing test standards?

The only correct answer here is to evaluate the test standards in comparison with what you are trying to get out of testing.

Example: if you are trying to have a defect-free embedded medical application (like a pace-maker), then your standards are probably too low. If you are writing prototype throw-away applications, like demos, your standards are definitely too high.

Another example: if your team is very senior, and you need to write lean and fast code, probably other forms of testing are more important than unit tests. If you have a junior team that is prone to make a lot of mistakes and you want to implement defensive standards, then this strategy doesn't seem strong enough.

That said, I would say that with the standards you are suggesting, you won't get a lot from the tests:

1. Partial code coverage is better than nothing, but if you introduce complexity due to testing you may have an increased defect count in the non-tested parts.

2. Partial code coverage does not allow your team to refactor mercilessly with impunity.

3. Giving developers time to write tests is good, but, asking them to write tests first is much better, and giving them acceptance tests is probably better in your case.

As you can see, my answers are very general, but they do represent my repeated experience. In no case mix-and-match indices have been useful to me.

There are two uses of code coverage, as far as I know:

• when your code has low coverage (say <85%), code coverage is only useful in identifying places that need testing love. The percentage is frankly irrelevant.

• when your code coverage is high, you can use a minimum coverage metric to turn the build red. Yes, high coverage doesn't guarantee correctness, but low coverage is certainly correlated with more defects, so once your team achieve high standards it's good if they can maintain them.

Your strategy of making the coverage check unobtrusive will only work if the team has already high code coverage, otherwise they will need to know exactly what to cover, in which case they will need to have constantly available a line-by-line map of the coverage they have!

Finally, if you have a lot of untested code simply giving your developers time to write tests for new features will not fix the initial coverage problems. This is important, because in order to make the new code testable, your team will need to refactor the old code, and in order to do that with confidence... they need to test the old code first. You haven't specified if this is for a greenfield project or a brownfield one - it makes a big difference.

---

Let me address the further information you provided.

• You want to shorten your feedback cycles;
• You want to motivate your team;
• You want your team to write tests;
• You want to reduce bug count for new features;
• You want to reduce regressions;

These objectives are laudable but very hard to achieve in an ideal situation, let alone in a complicated, legacy application scenario.

What I suggest is that you focus on the most important one and achieve that first, and then move on to the next. Make sure that once you start rearchitecting your application you have very well set architectural qualities (i.e., performance is a feature).

For example:

1. Put your testing and development team working together, and have them come back with a good plan to test the application automatically. The solution needs to be achievable within some constraints (typically of time). You don't want 100% coverage at this stage. You simply want an automate testing set-up to be run, with the quick-win tests being run on it.

2. Once the above is done, set some concrete objectives on quality, and let the team propose the best strategy to achieve these. Typically they will come up with some form of vertical testing, like integration tests. Measure everything, be transparent, monitor successes and failures. You want code coverage here, but also bug count graphs, etc.

3. Once the above is done, and if TDD is appropriate, refactor, rearchitecture and unit test the legacy code (wrap and refactor), and develop new features using TDD.

4. ...