Whlist browsing on github I came across the following project: https://github.com/infection/infection and according to the project's website https://infection.github.io/guide/#What-is-Mutation-Testing .

As far as I understood what a mutation testing mean is that automatically the code is slightly being changed and some tests are performed on it and the coverage is scored on a Mutation Indicator Score.

Practically unit tests tells us (explained in childish):

Hey you developed this nice function, I the Mrs Unit Test framework, if you explain to me how this specific functions is used (unit tests with intput mocking) and what does mean that works (assertions) I can check this out for you so in case you change stuff you can be sure that the code in not screwed up.

The integration testing tells us:

Hey I can tell you whether the software as a whole works together on a current setup. (eg. If the database fetching code with a given database fetches and processes the data as expected)

Also fuzzing test/ black box testing tells us:

Hey I can give you garbled input so you can check how the software works there are possible holes in harsh unpredicted conditions that you never thought of. So you can sleep bit quieter making sure that the software is safeish.

But what mutation testing tells us in practical terms, what MSI practically indicates to us the software engineers?

  • 4
    It's clearly stated in the link: "MSI can be used to measure the effectiveness of a test set in terms of its ability to detect faults." – jonrsharpe Feb 9 '19 at 14:37
  • 2
    Your "child-ish" text is less understandable (and a great deal less precise) than a brief, sensible description using ordinary sixth-grade English would be. – Robert Harvey Feb 9 '19 at 19:04

Mutation Testing runs your tests on slightly altered ("mutated") versions of your code. If your tests test the code thoroughly, then they should fail if you run them against a mutated version of your code. In particular, only a small number of tests that test the portion of the code that was mutated should fail.

Mutation Testing is in some sense Fuzz Testing for your Tests.

There are three scenarios:

  • The tests fail at the point the mutation was introduced: GOOD! Your test caught the modification.
  • The tests fail at some other point in the code: BAD! Your code is tightly coupled, and a change in one place affects some far-away unrelated code.
  • The tests don't fail. BAD! Your tests aren't thorough enough.

In the third case, there are actually two different possible underlying conditions:

  • Your tests are incomplete. Solution: add the missing test.
  • Your tests are complete, i.e. your tests test all the code you care about. In that case, the code that was mutated is obviously code that you don't care about (otherwise it would be covered by a test that fails when you mutate the code). Solution: just delete it!

The kinds of mutations the mutation tester performs depend on the specific mutation tester. The quality of a mutation tester lies mostly in how "intelligent" it is about applying mutations. (For example, if it were trying every single possible number for a long variable, it would never finish in time.)

Here are some examples of more basic mutations:

  • Changing literal values in the code:
    • Changing true to false or vice versa.
    • Changing 0 to non-zero or vice versa.
    • Changing a positive number to a negative number or vice versa.
    • Changing a large number to a small number or vice versa.
    • Changing a floating point number to NaN, +0,0, -0.0, +Inf, or -Inf or vice versa.
    • Changing a string to the empty string or vice versa.
    • Changing a large string to a small string or vice versa.
    • Changing an uppercase string to a lowercase string or vice versa.
    • Changing an array to the empty array or vice versa.
    • Changing a large array to a small array or vice versa.
    • Randomizing, sorting, or reverse sorting the elements of an array.
    • Changing a current date to a date in the distant past or future or vice versa.
    • Testing out known edge cases, e.g. 0, -1, 1, -2, 2, -3, 3, 127, 128, -128, -129, 254, 255, 2^32-1, 2^32, 2^32+1, 2^64-1, 2^64, 2^64+1, NaN, +0,0, -0.0, +Inf, -Inf, '', [], a string larger than 2^32 characters, a string larger than available physical memory, a string larger than available virtual memory, a string with non-alphanumeric characters, a string with non-ASCII characters, a string with non-BMP characters, various strings with sequences known to induce escaping errors (e.g. <, >, ;), an array larger than available physical memory, an array larger than available virtual memory, a sorted array, a reverse sorted array, a recursive array (i.e. an array that contains itself in languages where arrays are heterogeneous), an "unusual" date or time with leap days and leap seconds such as February 29th, or 23:59:60, and so on.
  • Changing variables / references
    • Changing defined references to null or vice versa
  • Changing subroutine arguments
    • Passing an argument for an optional parameter when none was passed or vice versa
    • Passing null when a value was passed
  • Changing control flow
    • Negating the condition of an if
    • Re-order the branches of a switch
    • throw an exception where none was thrown or vice versa
  • Changing statements
    • Replacing calls to rand with fixed values
    • Replacing calls to getTimeOfDay with fixed values
    • Exercising typical error paths for filesystem and network operations (network timeout, file not found, missing permissions)
    • Removing entire statements or even blocks altogether

Another important feature of a mutation tester is to know about the structure of your tests, so that it

  • knows to only run the tests that test the piece of code that it is currently mutating (to speed up the mutation testing) or
  • knows which test failures are relevant and which are due to tight coupling.

Metrics are a hard beast, just like with every other technology.

For example, you know why test coverage metrics don't actually tell what most people assume they tell you: which parts of your code are tested. In fact, they only tell you which part of your code was executed. If you have a test suite with perfect coverage, and you remove all assertions, then it will still run all the code it ran before, so it will still have 100% coverage, but it will actually test nothing. So, what test coverage really tells you is not which parts of the code are tested, but only which parts of the code are definitely not tested, namely the parts of the code that weren't even executed.

Likewise, Mutation Testing has its problems, too, which I alluded to above. If you mutate a piece of code, and a test fails that does not really tell you that your test caught the mutation. It could also be another test in a different part of the code that failed because your code is too tightly coupled. So, even if every mutation is accompanied by a failing test, you still need to examine the failing tests closely to verify that they actually failed due to the mutation.

OTOH, if you mutate a piece of code and no test fails, that doesn't 100% tell you that you are missing coverage. For example, maybe you are doing a case-insensitive string comparison, then mutating a string from lowercase to uppercase will not fail any test. So, you need to examine the "failing mutations" closely to verify that they actually should have failed a test.

  • In other words Mutations tests tells us how good are the other tests right? – Dimitrios Desyllas Feb 9 '19 at 15:18
  • @DimitriosDesyllas: In other words, it's more gooder. – Robert Harvey Feb 9 '19 at 19:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.