Unit testing and Test Driven Development questions

Question

I'm working on an ASP.NET MVC website which performs relatively complex calculations as one of its functions.

This functionality was developed some time ago (before I started working on the website) and defects have occurred whereby the calculations are not being calculated properly (basically these calculations are applied to each user which has certain flags on their record etc).

Note; these defects have only been observed by users thus far, and not yet investigated in code while debugging.

My questions are:

• Because the existing unit tests all pass and therefore do not indicate that the defects that have been reported exist; does this suggest the original code that was implemented is incorrect? i.e either the requirements were incorrect and were coded accordingly or just not coded as they were supposed to be coded?

• If I use the TDD approach, would I disgregard the existing unit tests as they don't show there are any problems with the calculations functionality - and I start by making some failing unit tests which test/prove there are these problems occuring, and then add code to make them pass?

Note; if it's simply a bug that is occurring that can be found while debugging the code, do the unit tests need to be updated since they are already passing?

Answer 1

Short Answer: Most probably unit tests are not well designed to cover the critical calculation scenarios. In addition, there might be business rule modifications/changes that are not covered in the existing unit tests, because of million reasons that one may face in a tightly scheduled development environment.

As was also mentioned:

Automatic unit testing doesn't imply that the final tests will cover all cases and code will be absolutely correct and bug-free. It is easy not to see some complex use-cases and possible bug.

There might be million reasons why it is NOT done right. My point is that you need to cover these cases with product owner, and then go through each unit test and verify the implementation by applying correct naming of each unit test, as well as by adding the missing once.

If I use the TDD approach, would I disgregard the existing unit tests as they don't show there are any problems with the calculations functionality - and I start by making some failing unit tests which test/prove there are these problems occuring, and then add code to make them pass?

TDD (Test-driven development) will for sure help, but i would NOT discard the current set of unit tests as far as they do what they supposed to be doing.

Answer 2

Using TDD and automatic unit testing doesn't imply that the final tests will cover all cases and code will be absolutely correct and bug-free. It is easy not to see some complex use-cases and possible bug.

TDD on the other hand makes it easier to introduce changes for those not-thought-before cases. Simply repeat TDD as if you were developing:

1. Write new test case, that fails
2. Implement code, that will make this test succeed
3. Refactor code to remove duplicities and introduce good abstractions while keeping all tests passing

Answer 3

This is exactly why you would want to consider Behavior driven development and Acceptance Test driven development - unit tests are not a guarantee about whether the system is doing what it is supposed to - it checks only whether it does what it was designed to do. The flip side is that product owners/Business analysts cannot review these tests very easily to verify that they are correct.

On the other hand, if you use acceptance tests, then they can be very easily verified by Business analysts or other non-tech folks. That can be quite useful.

In your particular case, I would say the product owner should take the decision on whether these are bugs or not - if they are, then obviously existing tests are either incorrect or at the very least insufficient - and you will have to write new (failing) ones and maybe even delete some of the old ones (which are expecting wrong things) before writing code that passes the new tests.

Answer 4

I do not believe that unit tests necessarily test requirements: 1) unit tests operate against very low level components (methods/classes/interfaces/et cetera) tested as stand-alone units; 2) requirements are supposed to be tested while the test engineer designs integration/system tests during the requirements design phase; 3) functionality, requirements and specifications are tested (again) once the system is in an operational state; and 4) using an iterative software development approach, many phases of the development lifecycle are revisited for tightening up requirements, design, and development -- processes that re-occur.

Why unit testing is not appropriate for testing requirements. Unit testing will not necessarily uncover a requirement issue for a number of reasons. One, if a requirement was poorly communicated, chances are the requirement was incorporated into the software design, and subsequent development; as a result, if the unit testing follows suit, correctly following an incorrect requirement, the unit testing will come up with nothing. Two, if the requirement was properly documented, and the development team misinterpreted the requirement, similar to item 1, the misintepretation will carry through to the unit testing. Three, as far my experience goes, unit testing is done by the developer, and is part of the development lifecycle -- unit testing is low-level and will not likely represent a requirement, unless each complete requirement is represented by its own method, which is very, very unlikely.

Example. Consider unit testing in terms of methods. Unit testing largely verifies that given the pre-condtions, post-conditions hold after execution of the method under test. The object of unit testing is to test/verify the methods, et cetera, of a software system, starting from the most atomic or primitive methods, progressing towards the most complex ones. By testing each low-level method, you verify whether it works, a no brainer. At this stage, the developer does not need to re-test the low-level method again as a stand-alone unit. While moving upward in complexity to higher-level methods, unit testing verifies these methods, demonstrating that the primitive methods, from which the higher-level ones are composed, work together. As far as I know, this is pretty much where unit testing stops.

How requirements are tested. In a well planned project, the test engineer (an integrated / system tester), starts designing tests while the requirements are written. Through this activity, requirements get verified, before system design, and long before a stitch of code is written. Once the system becomes available in some useable form, requirements are verified via functional/integration/system testing.