How to measure that a team is doing Test Driven Development

Question

My team is considering a department goal of doing Test Driven Development next (fiscal) year.

TDD sounds good and my manager is on board. The only problem is measurement.

Our company rules require that department goals be measurable.

Is there any (sane) way to measure if a team is doing TDD?

Note: We had a goal for writing unit tests already, so a code coverage goal will not work. We write the unit tests but we do it after the fact, not before.

Is there a way to measure that TDD is being done?

(NOTE: We use Visual Studio 2013 and TFS in case it matters)

Answer 1

Just like a unit test should test behaviour not implementation your department should measure the anticipated result of the policy, not implementation of the policy.

Your department wants to move to TDD. Why? What is it you're hoping to achieve through TDD? Fewer bugs raised that are regression related? Faster velocity on cases? Then you should measure these things instead of trying to find a way to measure that everyone is using TDD.

If the hoped for results of your push to TDD isn't something externally measurable, such as a more enjoyable developer experience and improved flow (one of the things I like about TDD), then you shouldn't need to change your testing at all because you'd be "refactoring" the department.

Instead I suggest just asking for feedback from the developers on how they think TDD is going. If they don't 'buy-in' to the TDD concept then it doesn't matter how you measure it - it will fail.

Answer 2

TDD is not something a team commits to doing. It's only a means to an end. Objectives can be writing more maintainable code, decreasing the number of defects, delivering faster, or improving test coverage, but not "doing TDD".

TDD is not a silver bullet. Measuring "if the team is doing TDD" does not guarantee success like making sure they drink their magic potion every day. Besides, demanding that the team produce indicators showing that they are practising a technique that the team itself asked for in the first place is a bit... absurd. Or command-and-control fundamentalistic.

Managers and decision makers shouldn't demand control over how code is produced at a micro level (a typical red/green/refactor cycle is the order of a few minutes). Since when do bean counters care about how the beans were harvested ?

Answer 3

Since you are going to measure unit test separately (code coverage, decision coverage, etc.), let me add a couple more tweaks to the process.

1. All check-ins must go through code review.
2. Code reviews should be trackable, i.e, should not be an over-the-shoulder review or like pair programming. You should be able to retrieve the code review comments, iterations and such details at a later time. TFS lets you do this.
3. When signing-off on a code review, the sign-off comments can a predetermined piece of text that tells TDD was followed. e.g.: "TDD FOLLOWED". This way you know which check-ins followed TDD, and at the end of the year, you can invent/calculate more metrics like percentage of LOC with TDD.

HOWEVER, I would not judge the value/effectiveness of TDD purely based on these metrics. I think the industry has evolved enough that the majority understands the value of TDD.

Answer 4

Since you want to switch from a test-second development to test driven design, you can best measure that by looking for proof that the tests were written before program was changed.

Teach your team to commit their failing tests first, so they can be measured, and then write the code to pass the test. You may need to either manually inspect the repository history, or write a custom tool to do such for you.

(You'll wind up treating it like a check-off for each project, so if check-in-first is needed you could always just have it be part of the reported information, and trust the programmers / project manager. i.e., "were the tests written before the code?")

Answer 5

If a team truly wants to do TDD, they you just need to do some minimum tracking to satisfy the rule. Check in your tests first and then check in the code and after comparing the initial dates, you have some semblance of proof.

To have another layer of approval, make this a part of the code review where someone can verify these are meaningful tests. It's not that hard to write a test that fails without even referencing the code it is suppose to test. It's just not in the spirit of what you're trying to do.

During some semblance of pair-programming, everyone can "sign-off" that TDD was going on.

You may not feel the need to take things this far, but you have to be prepared to serve the powers that be.

Personally, I think TDD should be used and evaluated to solve some larger problem. If releasing fewer bugs on your next upgrade is a concern because it lowers customer satisfaction, you can do a survey and ask customers: On a scale of 1-10 (10 being most painful) how much more painful was our latest upgrade (built on TDD) compared to the previous one? It's good enough for pain medication drug trials, is should be good enough for software.

Answer 6

To ensure team is following test driven development is to calculate DEFECT DENSITY.

You can agree mutually within the team/stakeholders to find this for the number of lines. We had 50 lines. That means, 1 bug per 50 lines will be quite ok to fix. So, if you want to keep 1000 lines as standard, then you should not exceed 20 bugs.

DEFECT DENSITY = Bugs found through Unit Testing/Lines of Code (1000 in my case). And if this came less than 20, we were ok to fix and move.

Answer 7

It's a stupid request proving the idiocy of many managers/accountants/cto etc who don't understand that software development is a kinda unique process in itself (so no, it doesn't matter that Toyota uses a specific process, you're not building cars)

Anyway, the only things that matters to be measured are RESULTS. Do you deliver higher quality software? Do you deliver it faster? Does the current method of developing (TDD or not) yields at least equal or better results with lower cost (time and money)?

It simply boils down to cost and results. You want better results with the same or lower costs or at least the same result with less costs. It's the ratio between the final result and the bottom line that matters.

Your team has to show that the TDD approach improves productivity (maintainability -> easier changes -> faster results -> less time required) and/or product quality (less bugs -> less time spent debugging -> reliable product | less people pissed off -> happier stakeholders).

I know that it takes more time to think and write tests but software devel doesn't stop when the first release is out. And even beancounters need to (forcefuly) understand that software development has its quirks, it's not an exact science and if they can do a better job, let them do it. You don't build houses how you build cars or airplanes, a cupcake business or cargo services. Why should you build software this way?

Edit: It might be news for some people but a lot of things simply CAN'T be measured reliably. You can pretend you're measuring it and come up with bullshit metrics but that's all. Marketing is one of the domains greatly affected by this plague.

Answer 8

I'm not sure that you'll need a different measure at all. You say:

We had a goal for writing unit tests already....

... presumably you are measuring this in some way. The fact that you now want to write tests before the code should have no effect on your existing metrics. After all, there may have been times when, (for example) to get a better understanding of the problem, someone in the team did write a test before the code. So in that case they were already practising TDD; they just didn't call it that, and management didn't care.

The existing goal is to have unit tests; you will still have unit tests after adopting TDD. It's simply a non-issue from a bean-counter perspective. As others have said, TDD in itself is not a goal.

Answer 9

TDD should generate at least 1 unit test per class. For every class in a project there should be at least 1 unit test that was written for that class, and it tests just that class.

Unit test to class ratios, code coverage, etc.. etc.. seem like things that give a meaningful measurement, but they don't.

What you want to measure is the complexity of classes, and how well those classes are tested. The higher the complexity of a class the greater the risk that class introduces into a system.

You can have 100% test coverage on a highly complex class, but that doesn't mean you've tested it well.

Complexity is the measurement of how many logical decision pathways there are in code. You can have a chunk of code that generates 1 million possible decision combinations and have 100% code coverage. It doesn't mean you tested all 1 million cases.

The other issue is how much of the system uses these complex class?

So this means some unit tests are more important than others, and it's those tests you want done very well.

This is something that is very hard to measure...