In a financial services environment - is it realistic to ask your team of ten programmers to take on a 100% automated unit and integration test coverage goal as a stretch task for a legacy java system (>1.5M SLOC)? [as a two year goal in increments]

  • 100% is a lot, and may require quite a bit of effort. – user1249 Jul 2 '11 at 11:07

It can be possible, but you have to ask, Is it worth it?

When doing normal TDD, you spend some time writing code that does not go into production, but the time is well spent because in the end you save more time because it is easier to add new features and you spend less time fixing bugs.

But when retrofitting unit tests to a legacy code base based on a goal of reaching 100% code coverage, the tests will often be written for the wrong reason. You may start out by writing a lot of tests based on the business logic. But after you reach a certain coverage (I would guess somewhere between 60-80%), you no longer write tests based on the business logic, but rather by looking at the code coverage analysis, finding lines of code that are not covered.

The tests covering the last 20-40% of have a high cost, a few reasons:

  • They have no real value
  • It takes an immense amount of time to write. When tests are based on business rules, then a single test will typically cover many lines of code. Once you start writing tests based on code coverage analysis, you tend to write one test for every single line of uncovered code.
  • The tests tends to have to be refactored very often.

What you must also realize is that a system that does not have any test coverage is probably not easy to test to begin with, so just undertaking such an endavour, retrofitting unit tests to an existing code base, the existing code base has to be refactored quite heavily.

So is it realistic? Perhaps. Is it a good idea? Not likely.

| improve this answer | |

No, it's not realistic. It can be done, but will take much longer, cost much more, and do very little more (or even less) for project maintainability than a smarter goal of e.g. 80% on only those parts of the system that have been identified as high-risk.

The problem is that a system that has not been designed to have automated tests from the start is probably not very tractable to testing and can require a lot of refactoring to get there, which is a very risky activity. And this problem will be magnified a hundredfold of the goal is to get 100% coverage (not even touching on the subject of tests that increase coverage but don't actually test anything and are therefore useless).

Basically, when it comes to automated testing, code can be measured on two orthogonal axes:

  • Easy to test vs. hard to test
  • Complex, frequently-changing code that benefits a lot from testing vs. simple, rock-solid code that doesn't really need testing

The position of code on these axes translates directly to a cost-benefit ratio for tests. Easy-risky code is where you should start, probably followed by hard-risky (if quality is most important and you have enough time) or easy-solid (if cost matters or there is deadline pressure). Hard-solid code should be tackled last, in the hope that the by then existing tests will make the necessary refactorings less risky, or just left untested because it's just not worth it.

After all, you still need manual tests anyway, since not everything can be automated and not every unexpected failure mode be caught by automated tests.

| improve this answer | |
  • As a QA this is the exact approach that I take to legacy applications with no test automation. – Lyndon Vrooman Jul 2 '11 at 13:17

Assuming you mean the entire code base and not just what they're working on, it sounds like a somewhat bizarre goal to me - why is the business interested in this? Is it going to generate them more money? And if the aim is to make the code base more resiliant to change then presumably most changes are only going to affect a small proportion of those 1.5M LOC and the time spent adding coverage to the rest is simply wasted.

Better would be to introduce tests as you go, isolating the code that will change from that which won't, together with some throw-away large scale integration tests when you're starting.

The book Dealing Effectively with Legacy Code has excellent advice in this area.

| improve this answer | |

As a programmer on such a project I'd have a few reservations.

Firstly, why are we undertaking such a project. Are there planned additions to the project, or is there time being scheduled for a major rework. Testing for the sake of testing is a poor goal and will kill motivation. If there is a strong business goal behind the project perhaps I'd be happy to do it.

Secondly, once I understand the goal, is coverage really the best manner in which we can achieve this goal. There are numerous criticisms about test LOC coverage. Which ones apply here? Will we simply be wasting time with that final 10-20%. Is there a way we can partition the project to incrementally achieve our primary goal?

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.