Generally when writing automated unit tests (eg JUnit, Karma) I aim to:

  • cover all the boundary conditions
  • get a high level of coverage.

I heard someone say:

coverage and boundary conditions aren't enough for a unit test, you need to write them so they will break if the code changes.

This sounds good to me in theory - but I'm not sure how to apply it.

My question is: Should I write automated unit tests that fail when the code changes? If so, how.

  • 1
    where did you hear that and how exactly it was stated? (FWIW as quoted it doesn't make sense - you likely misunderstood)
    – gnat
    Commented Jan 9, 2016 at 11:15
  • 36
    Automated tests aren't meant to break when code changes. They're meant to ensure that when you change code that your code still works to spec, and if it doesn't THEN fail.
    – Pieter B
    Commented Jan 9, 2016 at 11:18
  • 3
    There is a reason for a code to fail. That is when a procedure or business logic changes. Unit tests cannot predict domain changes, so if you do that, then the tests should fail and they should be rewritten. If you, however, only refactor code to make it faster, better (, stronger?), then all your unit tests should work even after you are done with the changes.
    – Andy
    Commented Jan 9, 2016 at 11:22
  • 3
    That edit completely changes the meaning of the question, by the way. I see three answers, and they've all been invalidated. Commented Jan 10, 2016 at 12:41
  • 2
    @BlacklightShining: There are actually six answers; three, including mine, have been deleted. This question should have been allowed to die with dignity instead of edited.
    – Blrfl
    Commented Jan 10, 2016 at 13:20

2 Answers 2


Your aim should be not to write unit tests that fail when the code changes, but unit tests that fail when the behaviour changes. Here, behaviour means anything that an external caller of the method wants it to do, like returning the right response to a question or saving the right thing to a databse. How it achieves that is its own internal implementation, not its behaviour.

By testing behaviour rather than implementation, you can refactor code to make improvements, and instantly verify whether you've accidentally changed the way it behaves by running your tests.

In reality, it's not possible to perfectly achieve this goal. If you have a method:

int add(int x, int y) {
    return x + y;

You can write as many unit tests as you want for it, but it's extremely unlikely any of them will fail if you modify it to:

int add(int x, int y) {
    if(x==10731 && y == -405571) {
        return 0;
    return x + y;

However, you can take some sensible steps to get as close to full behavioural coverage as is practical:

  • As you said, think about boundary conditions and corner cases. These are the places where you're most likely to see accidental behaviour change.
  • Think of line-by-line coverage as "necessary, but not sufficient". Imagine trying to be as lazy as you possibly can while still getting full line-by-line coverage, and you'll see how easy it is to write an inadequate set of tests that follow this
  • Think about the behaviour your method is supposed to provide, and the branches it can follow. Ideally you should test that for every route through its implementation, it provides all the behaviour that's expected.
  • When you've written a set of tests, ask "Are there any implementations of my method which are at least as simple as the existing one, which would pass all of my tests, but which have the wrong behaviour?" This is a good rule of thumb to see if your behavioural coverage is good enough.

    As shown in the example add method above, you can never really fully defend against modifications to your method that add extra stuff to it, making it less simple. But much more likely, bugs are going to sneak in through modifying or removing parts, without adding to the complexity (because, why would you refactor something in a way that adds to its complexity?). So by adding the "at least as simple" condition, you get something practically achievable.

  • 3
    I will note that your example bug would be caught if you ensure 100% branch coverage when running your tests. Such tests might not need to be written by hand. Even an automated tool like (C#) Pex (e.g., paste this code into Pex4Fun) would catch this type of bug, though I'm unsure what sort of tools exist for Java.
    – Brian
    Commented Jan 9, 2016 at 16:25
  • @Brian Agreed. There's also things like property-based tests that can make special cases at least more likely to be caught. Nothing's foolproof though, and I didn't want to muddy the answer with increasingly advanced ways to plug an increasingly small hole Commented Jan 9, 2016 at 17:48
  • This is a really helpful constructive answer. I've updated the quote - would you consider updating your question?
    – hawkeye
    Commented Jan 10, 2016 at 11:02
  • @Brian: And let's not forget branch-aware fuzzing, AFL or the LLVM's libFuzzer are examples of libraries that observe the code behavior at run-time and try to generate input that triggers new code paths. Commented Jan 10, 2016 at 12:23
  • Fuzzing is interesting - I'm told the tool that specifically addresses this problem is a mutation test tool: mutation-testing.org
    – hawkeye
    Commented Jan 10, 2016 at 12:27

The answers given - that this is a wrongheaded thing to do, and violates every principle of good testing - are correct.

But this is programming. There are always some sensible rationales for even the strangest request, and it can be productive to consider them.

Say you have some critical kernel code, or some high-security part of your SSH library, which absolutely, under no circumstances, should ever change its behavior, not even for some tiny subset of the infinite values it takes, all of which you could never test against.

And say you have a company set up such that there is a QA department responsible for writing unit tests against code provided by the Dev deparment, to ensure it meets business requirements.

In this situation, it makes sense to say "the critical code paths should only be modified by Dev when QA has been notified with signoff by upper management." Any change without signoff and notification is non-deliberate, and may be malicious.

In that case, a test which uses introspection to get a hash of the contents of each method, and compare that to a stored hash, could be legitimate. You can't serialize Method(), but you could subclass Method to be serializable. Perhaps a better approach is to get a hash of the file itself. Or just compare the file to a secure backup of the file.

But probably the best approach if the requirement is to fail on any modification, is to check the versioning system's logs and see if any changes were downloaded for the relevant files. If yes, then fail.

  • "QA department responsible for writing unit tests" Please, no.
    – Celos
    Commented Jan 15, 2016 at 8:03
  • @Celos - Coders should write low-level unit tests as they code. But another layer of higher level tests written by people who are not the people writing the code exposes a plethora of spec issues that you simply can't ever through coder testing. All too many places leave spec-compliance-testing to some combo of [arbitrary coder decision; manual QA tests; wishful thinking]. Like art and UI design, automated testing's a separate concern and specialization from coding, and most coders who think that they can do these things just fine, simply haven't a clue. Commented Jan 16, 2016 at 0:36
  • In the case where sign-off to the code is required, you shouldn't use tests for that. There are better tools for that job, ones that operate on an entirely different level from unit tests.
    – Jasmijn
    Commented Jun 10, 2022 at 12:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.