Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
Generally when writing automated unit tests (eg JUnit, Karma) I aim to:
I heard someone say:
coverage and boundary conditions aren't enough for a unit test, you need to write them so they will break if the code changes.
This sounds good to me in theory - but I'm not sure how to apply it.
My question is: Should I write automated unit tests that fail when the code changes? If so, how.
Your aim should be not to write unit tests that fail when the code changes, but unit tests that fail when the behaviour changes. Here, behaviour means anything that an external caller of the method wants it to do, like returning the right response to a question or saving the right thing to a databse. How it achieves that is its own internal implementation, not its behaviour.
By testing behaviour rather than implementation, you can refactor code to make improvements, and instantly verify whether you've accidentally changed the way it behaves by running your tests.
In reality, it's not possible to perfectly achieve this goal. If you have a method:
int add(int x, int y) {
return x + y;
}
You can write as many unit tests as you want for it, but it's extremely unlikely any of them will fail if you modify it to:
int add(int x, int y) {
if(x==10731 && y == -405571) {
return 0;
}
return x + y;
}
However, you can take some sensible steps to get as close to full behavioural coverage as is practical:
When you've written a set of tests, ask "Are there any implementations of my method which are at least as simple as the existing one, which would pass all of my tests, but which have the wrong behaviour?" This is a good rule of thumb to see if your behavioural coverage is good enough.
As shown in the example add method above, you can never really fully defend against modifications to your method that add extra stuff to it, making it less simple. But much more likely, bugs are going to sneak in through modifying or removing parts, without adding to the complexity (because, why would you refactor something in a way that adds to its complexity?). So by adding the "at least as simple" condition, you get something practically achievable.
The answers given - that this is a wrongheaded thing to do, and violates every principle of good testing - are correct.
But this is programming. There are always some sensible rationales for even the strangest request, and it can be productive to consider them.
Say you have some critical kernel code, or some high-security part of your SSH library, which absolutely, under no circumstances, should ever change its behavior, not even for some tiny subset of the infinite values it takes, all of which you could never test against.
And say you have a company set up such that there is a QA department responsible for writing unit tests against code provided by the Dev deparment, to ensure it meets business requirements.
In this situation, it makes sense to say "the critical code paths should only be modified by Dev when QA has been notified with signoff by upper management." Any change without signoff and notification is non-deliberate, and may be malicious.
In that case, a test which uses introspection to get a hash of the contents of each method, and compare that to a stored hash, could be legitimate. You can't serialize Method(), but you could subclass Method to be serializable. Perhaps a better approach is to get a hash of the file itself. Or just compare the file to a secure backup of the file.
But probably the best approach if the requirement is to fail on any modification, is to check the versioning system's logs and see if any changes were downloaded for the relevant files. If yes, then fail.
