What's the best way to organize our unit tests

Question

We've built up a substantial number of unit tests for our main program over the years. Several thousand. The problem is that we don't have a clear idea of what tests we have because there are so many. And that's a problem because we don't know where we're weak on tests (or where we have duplicates).

Our app is a reporting engine. So you can have a template that is used to test parsing (do we read all table properties), merging in data (did we keep the correct table properties in the merge), formatting the final page (is the table placed correctly on the page), and/or the output format (is the created DOCX file correct).

Add to this what we need to test. Take the padding around a table cell (we use Word, Excel, & PowerPoint for the report design). We have to test the padding across page break, for a table inside a cell, vertically merged cells, horizontally merged cells, a vertically & horizontally merged cell that contains a table with a vertically & horizontally merged cells in the inner table, where that table breaks across a page.

So what category does that test go in? Table padding, page breaks, nested cells, vertically merged cells, horizontall merged cells, or something else?

And how do we document these categories, name the unit tests, etc.?

Update: A number of people have suggested using coverage tools to verify that we have full coverage. Unfortunately that is of limited use in our case because the bugs tend to be due to specific combinations so it's code that has all been tested, but not in that combination.

For example, we had a customer yesterday that started a Word bookmark at the end of a forEach loop in their template (a Word document) and ended it in the beginning of the next forEach loop. This all used code that has unit tests against it, but we had not thought of the combination of a template expanding a bookmark start to be started 25 times, then ended 10 times (the two forEach loops had a different number of rows).

Answer 1

Generally, I tend to mirror the source tree for my unit tests. So, if I had src/lib/fubar, I would have a test/lib/fubar which would contains the unit tests for fubar.

However, what you seem to be describing are more functional tests. In that case, I would have a multi-dimensional table that enumerated all your possible conditions. Then, the ones that have no tests are either nonsensical or need a new test. You can, of course, then put them in sets of test suites.

Answer 2

The most common structure seems to be the src and test directory mirror.

src/module/class
test/module/class_test

However, there is an alternative structure that I have seen which I now believe to be better.

src/module/class
src/module/class_test

Since unit test tends to mirror the class they're testing, placing them in the same directory gives much easier access to the files so you can work on both side by side.

Answer 3

In .NET, I tend to mirror, or at least approximate, the namespace structure for the source code in the test projects, underneath a "Tests.Unit" or "Tests.Integration" master namespace. All unit tests go in one project, with the basic structure of the source code replicated as folders within the project. Same for integration tests. So, a simple solution for a project might look like this:

Solution
   MyProduct.Project1 (Project)
      Folder1 (Folder)
         ClassAA (Class def)
         ...
      Folder2
         ClassAB
         ...
      ClassAC
      ...
   MyProduct.Project2
      Folder1
         ClassBA
         ...
      ClassBB
      ...
   ...
   MyProduct.Tests.Unit
      Project1
         Folder1
            ClassAATests
            ClassAATests2 (possibly a different fixture setup)
         Folder2
            ClassABTests
         ClassACTests
      Project2
         Folder1
            ClassBATests
         ClassBBTests
      ...
   MyProduct.Tests.Integration
      Project1 (a folder named similarly to the project)
         Folder1 (replicate the folders/namespaces for that project beneath)
            ClassAATests
         Folder2
            ClassABTests
         ClassACTests
      Project2
         Folder1
            ClassBATests
         ClassBBTests

For any AATs or AEETs that are coded with a unit-testing framework, this changes a bit; usually those tests mirror a set of steps, that will test the functionality of a new use case or story. I usually structure these tests in a MyProduct.Tests.Acceptance project as such, with tests for each story, possibly grouped by milestone or "epic" story which the story under development belonged to. However, those are really just uber-integration tests, and so if you prefer to structure the tests in a more object-oriented instead of story-oriented fashion you don't even need a MyProduct.Tests.Acceptance or similar project; just throw em in MyProduct.Tests.Integration under the scope of the highest-level object under test.

Answer 4

There is no reason for a unit test to be only in one category. All major unit test toolkits support the creation of test suites, which bundle tests for a particular category. When a particular area of code has been changed, the developer should run that suite first and often to see what has broken. When a test concerns padding and breaks and nesting, by all means put it into all three suites.

That said, the point of unit tests is to run them all the time, i.e. they should be small and fast enough that it's feasible to run them all befor committing any code. In other words, it doesn't really matter what category a test is, it should be run before committing anyway. Suites are really just a crutch that you use if for some reason you can't write tests that are as fast as they should.

As for coverage, there are very good coverage tools that tell you what percentage of lines were actually exercised by running your tests - this is an obvious pointer to what kind of tests you are still missing.

As for naming, there is no particular value in expending effort on the names of unit tests. All that you want to hear from your tests is "5235 of 5235 tests passed". When a test fails, what you read is not its name, but the message, e.g. the String in the assert() that implements your success critrion. The message should be informative enough that you have an idea of what's wrong without reading the body of the test. The name is unimportant compared to that.

Answer 5

One way of knowing whether you are weak on tests is traceability. Usually for tests, this takes the form of coverage.

The aim is to measure what parts of code are exercized by your tests, so that you can see code that is not covered by your tests. It is up to you (and the coverage tool) to define what a "part of code" is. The least is branch coverage.