We have written close to 3,000 tests -- data has been hard coded, very little reuse of code. This methodology has began to bite us. As the system changes we find ourselves spending more time fixing broken tests. We have unit, integration and functional tests.

What I'm looking for is a definitive way to write manageable and maintainable tests.


  • This is much better suited to Programmers.StackExchange, IMO ...
    – IAbstract
    Commented Sep 20, 2011 at 19:07
  • BDD
    – Robbie Dee
    Commented Oct 8, 2015 at 8:16

8 Answers 8


Don't think of them as "broken unit tests", because they are not.

They are specifications, which your program no longer supports.

Don't think of it as "fixing the tests", but as "defining new requirements".

The tests should specify your application first, not the other way around.

You can't say you have a working implementation until you know it works. You can't say it works until you test it.

A few other notes that might guide you:

  1. The tests and the classes under test should be short and simple. Each test should only check for a cohesive piece of functionality. That is, it doesn't care about things which other tests already check.
  2. The tests, and your objects, should be loosely coupled, in a way that if you change an object, you're only changing its dependency graph downwards, and other objects which use that object are not affected by it.
  3. You might be creating and testing the wrong stuff. Are your objects built for easy interfacing, or easy implementation? If it's the latter case, you're going to find yourself changing a lot of code that uses the old implementation's interface.
  4. In the best case, strictly adhere to the Single Responsibility principle. In the worse case, adhere to the Interface Segregation principle. See SOLID Principles.
  • 7
    +1 for Don't think of it as "fixing the tests", but as "defining new requirements".
    – StuperUser
    Commented Sep 21, 2011 at 10:08
  • 3
    +1 The tests should specify your application first, not the other way around
    – treecoder
    Commented Sep 27, 2011 at 7:50

What you describe may actually not be such a bad thing, but a pointer to deeper problems your tests discover

As the system changes we find ourselves spending more time fixing broken tests. We have unit, integration and functional tests.

If you could change your code, and your tests would not break, that would be suspicious to me. The difference between a legitimate change and a bug is only the fact that it is requested, an what is requested is (TDD assumed) defined by your tests.

data has been hard coded.

Hard coded data in tests is imho a good thing. Tests work as falsifications, not as proofs. If there is too much calculation, your tests may be tautologies. For example:

assert sum([1,2,3]) == 6
assert sum([1,2,3]) == 1 + 2 + 3
assert sum([1,2,3]) == reduce(operator.add, [1,2,3])

The higher the abstraction, the closer you get to the algorithm, and by that, closer to comparing the acutal implementation to itself.

very little reuse of code

The best reuse of code in tests is imho 'Checks', as in jUnits assertThat, because they keep the tests simple. Besides that, if the tests can be refactored to share code, the actual code tested likely can be too, thus reducing the tests to the ones testing the refactored base.

  • I'd like to know where the downvoter disagrees.
    – keppla
    Commented Sep 22, 2011 at 6:40
  • keppla - I'm not the downvoter, but generally, depending on where I'm at in the model, I favor testing object interaction over testing data at the unit level. Testing data works better at an integration level. Commented Sep 27, 2011 at 1:29
  • @keppla I have a class which routes an order to a different channel if its total items contain certain restricted items. I create a fake order populate it with 4 items two of which are the restricted ones. In as far as the restricted items are added this test is unique. But the steps of creating a fake order, and adding two regular items is the same setup which another test uses which tests non-restricted item workflow. In this case along with items if the order needs to have customer data setup and addresses setup etc isnt this good case of reusing setup helpers. Why only assert reuse? Commented Oct 30, 2017 at 12:00

I've had this problem as well. My improved approach has been as follows:

  1. Don't write unit tests unless they're the only good way to test something.

    I am fully prepared to admit that unit tests have the lowest cost of diagnosis and time-to-fix. This makes them a valuable tool. The problem is, with the obvious your-mileage-may-vary, that unit tests are often too petty to merit the cost of maintaining the code mass. I wrote an example at the bottom, have a look.

  2. Use assertions wherever they are equivalent to the unit test for that component. Assertions have the nice property that they are always verified throughout any debug build. So instead of testing the "Employee" class constraints in a separate unit of tests, you're effectively testing the Employee class through every test case in the system. Assertions also have the nice property that they do not increase the code mass as much as unit tests (which eventually require scaffolding/mocking/whatever).

    Before someone kills me: production builds should not crash on assertions. Instead, they should log at the "Error" level.

    As a caution to someone who hasn't thought about it yet, do not assert anything on user or network input. It is a huge mistake™.

    In my latest code bases, I've been judiciously removing unit tests wherever I see an obvious opportunity for assertions. This has significantly lowered the cost of maintenance overall and has made me a much happier person.

  3. Prefer system/integration tests, implementing them for all of your primary flows and user experiences. Corner-cases probably do not need to be here. A system test verifies behavior at the user-end by running all components. Because of that, a system test is necessarily slower, so write the ones that matter (no more, no less) and you will catch the most important problems. System tests have very low maintenance overhead.

    It's key to remember that, since you're using assertions, each system test will run a couple hundred "unit tests" at the same time. You're also rather assured that the most important ones get run multiple times.

  4. Write strong APIs that can be tested functionally. Functional tests are awkward and (let's face it) kind of meaningless if your API makes it too hard to verify functioning components on their own. Good API design a) makes testing steps straightforward and b) begets clear and valuable assertions.

    Functional testing is the hardest thing to get right, especially when you have components communicating one-to-many or (even worse, oh god) many-to-many across process barriers. The more inputs and outputs attached to a single component, the harder functional testing is, because you have to isolate one of them to really test its functionality.

On the issue of "don't write unit tests," I will present an example:

    InternalDataStructureType sink;
    ASSERT_THROWS(sink.consumeFrom(NULL), std::logic_error);
    try {
    } catch (const std::logic_error& e) {
        ASSERT(e.what() == "You must not pass NULL as a parameter!");

The writer of this test has added seven lines that do not contribute at all to the verification of the final product. The user should never see this happening, either because a) nobody should ever be passing NULL there (so write an assertion, then) or b) the NULL case should cause some different behavior. If the case is (b), write a test that actually verifies that behavior.

My philosophy has become that we should not test implementation artifacts. We should only test anything that can be considered an actual output. Otherwise, there's no way to avoid writing twice the basic mass of code between the unit tests (which force a particular implementation) and the implementation itself.

It is important to note, here, that there are good candidates for unit tests. In fact, there are even several situations where a unit test is the only adequate means by which to verify something and in which it is of high value to write and maintain those tests. Of the top of my head, this list includes nontrivial algorithms, exposed data containers in an API, and highly-optimized code that appears "complicated" (a.k.a. "the next guy will probably screw it up.").

My specific advice to you, then: Start deleting unit tests judiciously as they break, asking yourself the question, "is this an output, or am I wasting code?" You will probably succeed in reducing the number of things that are wasting your time.

  • 4
    Prefer system/integration tests - This is mind-numbingly bad. Your system gets to the point where its using these (sloww!) tests to test the things that could be caught quickly at a unit level and it takes hours for them to run because you have so many similar and slow tests. Commented Sep 27, 2011 at 1:49
  • 1
    @Adres Jaan Tack - Asserts should be banned. They generate different code paths in debug vs release, they break automated testing environments, and it keeps the assert writing developer from actually handling the issue. Commented Sep 27, 2011 at 11:22
  • 1
    @RitchMelton Entirely separate from the discussion, it sounds like you need a new CI server. CI should not behave like that. Commented Sep 29, 2011 at 21:41
  • 1
    A crashing program (which is what assertions do) should not kill your test runner (CI). That's why you have a test runner; so something can detect and report such failures. Commented Sep 29, 2011 at 22:02
  • 1
    The debug-only 'Assert'-style assertions that I'm familiar with (not test assertions) pop up a dialog that hangs the CI because it is waiting for developer interaction. Commented Sep 29, 2011 at 22:13

Seems to me like your unit testing works like a charm. It is a good thing that it is so fragile to changes, since that's sort of the whole point. Small changes in code break tests so that you can eliminate the possibility of error throughout your program.

However, keep in mind that you only truly need to be testing for conditions which would make your method fail or give unexpected results. This would keep your unit testing more prone to "break" if there is a genuine problem rather than trivial things.

Though it seems to me that you're heavily redesigning the program. In such instances, do whatever you need to and remove the old tests and replace them with new ones afterwards. Repairing unit tests is only worthwhile if you're not fixing due to radical changes in your program. Otherwise you may find that you're dedicating too much time towards rewriting tests to be applicable in your newly written section of program code.


Your unit tests are biting you because you are following the traditional approach to testing, as described "by the book". As you can see, many answers are telling you that everything is fine and this is how it is supposed to be. In other words, most people find no fault with this way of unit testing.

I am here to say otherwise.

The traditional approach to testing (unit testing with mocks) is testing against the implementation, not against the interface. Thus, it is white-box testing by nature. As you have already found out, this is very highly problematic and it is bound to bite you.

The solution to this problem is to always test against the interface, not against the implementation. In other words, to practice black-box testing. Mocking frameworks, no matter how nifty and ingenious they are, force white-box testing, therefore they should be avoided like covid-19.

You are not limited to mocks and stubs; you can also use fakes. A fake supports the full functionality of the real thing, (or at any rate, the part of the functionality that we have a use for,) but is much more lightweight than the real thing, and it achieves this by making some severe compromise, such as not persisting anything to disk, which is irrelevant when testing anyway. For example, an in-memory database is a fake that you can use in place of an actual RDBMS.

Once you start making use of fakes, you can start doing integration testing only, as Andres Jaan Tack suggested in his answer. Then, your tests can be black-box tests. Black-box tests are reusable, so if you have two different implementations of the same interface you can use one test to test them both. This comes especially handy if you keep building both real modules and their fakes, because you can test both the fake and the real thing with the same test.

The beautiful thing about a fake is that it is a self-contained module which implements a known abstraction, so it is easy to maintain, whereas every single instance of mocking code scattered throughout the test codebase is an instance of incidental complexity in the tests, and therefore hard to maintain.

For more on this idea, you can read this post: michael.gr - Incremental Integration Testing

Disclosure: I am the author of that post.

  • 1
    Thank you for posting this answer to my question more than a decade after I asked it. I found your answer insightful and useful. Commented Nov 18, 2022 at 9:27

I am sure others will have a lot more input, but in my experience, these are some important things that will help you:

  1. Use a test object factory to build input data structures, so you don't need to duplicate that logic. Perhaps look into a helper library such as AutoFixture to cut down on the code needed for test setup.
  2. For each test class, centralize creation of the SUT, so it will be easy to change when things get refactored.
  3. Remember, that test code is just as important as production code. It should also be refactored, if you find that you are repeating yourself, if the code feels unmantainable, etc, etc.
  • The more you reuse code across tests, the more fragile they become, because now changing one test can break another. That may be a reasonable cost, in return for maintainability - I'm not getting into that argument here - but to argue that points 1 and 2 make tests less fragile (which was the question) is just wrong.
    – pdr
    Commented Sep 21, 2011 at 8:29
  • @driis - Right, test code has different idioms than running code. Hiding things by refactoring 'common' code and using stuff like IoC containers just masks design problems being exposed by your tests. Commented Sep 27, 2011 at 1:51
  • While the point @pdr makes is likely valid for unit tests, I'd argue that for integration/system tests, it might be useful to think in terms of "prepare the application for task X". That might involve navigating to the proper place, setting certain run-time settings, open a data file, and so on. If multiple integration tests start in the same place, refactoring that code to reuse it across multiple tests might not be a bad thing if you understand the risks and limitations of such an approach.
    – user
    Commented Sep 27, 2011 at 6:35

Handle tests like you do it with source code.

Version control, checkpoint releases, issue tracking, "feature ownership", planning and efforts estimation etc etc. Been there done that - I think this is the most efficient way to deal with problems like you describe.


You should definitely have a look at Gerard Meszaros's XUnit test patterns. It has a great section with many recipes to reuse your test code and avoid duplication.

If your tests are brittle, it could also be that you don't resort enough to test doubles. Especially, if you recreate whole graphs of objects at the beginning of each unit test, the Arrange sections in your tests might become oversized and you might often find yourself in situations where you have to rewrite the Arrange sections in a considerable number of tests just because one of your most commonly used classes has changed. Mocks and stubs can help you here by cutting down on the number of objects you have to rehydrate to have a relevant test context.

Taking the unimportant details out of your test setups via mocks and stubs and applying test patterns to reuse code should reduce their fragility significantly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.