Mutation Testing runs your tests on slightly altered ("mutated") versions of your code. If your tests test the code thoroughly, then they should fail if you run them against a mutated version of your code. In particular, only a small number of tests that test the portion of the code that was mutated should fail.
Mutation Testing is in some sense Fuzz Testing for your Tests.
There are three scenarios:
- The tests fail at the point the mutation was introduced: GOOD! Your test caught the modification.
- The tests fail at some other point in the code: BAD! Your code is tightly coupled, and a change in one place affects some far-away unrelated code.
- The tests don't fail. BAD! Your tests aren't thorough enough.
In the third case, there are actually two different possible underlying conditions:
- Your tests are incomplete. Solution: add the missing test.
- Your tests are complete, i.e. your tests test all the code you care about. In that case, the code that was mutated is obviously code that you don't care about (otherwise it would be covered by a test that fails when you mutate the code). Solution: just delete it!
The kinds of mutations the mutation tester performs depend on the specific mutation tester. The quality of a mutation tester lies mostly in how "intelligent" it is about applying mutations. (For example, if it were trying every single possible number for a
long variable, it would never finish in time.)
Here are some examples of more basic mutations:
- Changing literal values in the code:
false or vice versa.
0 to non-zero or vice versa.
- Changing a positive number to a negative number or vice versa.
- Changing a large number to a small number or vice versa.
- Changing a floating point number to
-Inf or vice versa.
- Changing a string to the empty string or vice versa.
- Changing a large string to a small string or vice versa.
- Changing an uppercase string to a lowercase string or vice versa.
- Changing an array to the empty array or vice versa.
- Changing a large array to a small array or vice versa.
- Randomizing, sorting, or reverse sorting the elements of an array.
- Changing a current date to a date in the distant past or future or vice versa.
- Testing out known edge cases, e.g.
, a string larger than 2^32 characters, a string larger than available physical memory, a string larger than available virtual memory, a string with non-alphanumeric characters, a string with non-ASCII characters, a string with non-BMP characters, various strings with sequences known to induce escaping errors (e.g.
;), an array larger than available physical memory, an array larger than available virtual memory, a sorted array, a reverse sorted array, a recursive array (i.e. an array that contains itself in languages where arrays are heterogeneous), an "unusual" date or time with leap days and leap seconds such as February 29th, or 23:59:60, and so on.
- Changing variables / references
- Changing defined references to
null or vice versa
- Changing subroutine arguments
- Passing an argument for an optional parameter when none was passed or vice versa
null when a value was passed
- Changing control flow
- Negating the condition of an
- Re-order the branches of a
- throw an exception where none was thrown or vice versa
- Changing statements
- Replacing calls to
rand with fixed values
- Replacing calls to
getTimeOfDay with fixed values
- Exercising typical error paths for filesystem and network operations (network timeout, file not found, missing permissions)
- Removing entire statements or even blocks altogether
Another important feature of a mutation tester is to know about the structure of your tests, so that it
- knows to only run the tests that test the piece of code that it is currently mutating (to speed up the mutation testing) or
- knows which test failures are relevant and which are due to tight coupling.
Metrics are a hard beast, just like with every other technology.
For example, you know why test coverage metrics don't actually tell what most people assume they tell you: which parts of your code are tested. In fact, they only tell you which part of your code was executed. If you have a test suite with perfect coverage, and you remove all assertions, then it will still run all the code it ran before, so it will still have 100% coverage, but it will actually test nothing. So, what test coverage really tells you is not which parts of the code are tested, but only which parts of the code are definitely not tested, namely the parts of the code that weren't even executed.
Likewise, Mutation Testing has its problems, too, which I alluded to above. If you mutate a piece of code, and a test fails that does not really tell you that your test caught the mutation. It could also be another test in a different part of the code that failed because your code is too tightly coupled. So, even if every mutation is accompanied by a failing test, you still need to examine the failing tests closely to verify that they actually failed due to the mutation.
OTOH, if you mutate a piece of code and no test fails, that doesn't 100% tell you that you are missing coverage. For example, maybe you are doing a case-insensitive string comparison, then mutating a string from lowercase to uppercase will not fail any test. So, you need to examine the "failing mutations" closely to verify that they actually should have failed a test.