How would type errors be detected while creating mocks in a dynamic language?

Question

The problem occurs while doing TDD. After a couple of test pass, the return types of some class/module change. In a statically typed programming language, if a previous mocked object was used in the tests of some other class and was not modified to reflect the type change, then compilation errors will occur.

For dynamic languages however, the change in return types might not be detected and the tests of the other class will still pass. Sure there might be integration tests that should fail later on, but unit tests would erroneously pass. Is there any way how to avoid this?

Updating with a trivial sample (on some made up language)...

Version 1:

Calc = {
    doMultiply(x, y) {return x * y}
}
//.... more code ....

// On some faraway remote code on a different file
Rect = {
    computeArea(l, w) {return Calc.doMultipy(x*y)}
}

// test for Rect
testComputeArea() { 
    Calc = new Mock()
    Calc.expect(doMultiply, 2, 30) // where 2 is the arity
    assertEqual(30, computeArea)
}

Now, on version 2:

// I change the return types. I also update the tests for Calc
Calc = {
    doMultiply(x, y) {return {result: (x * y), success:true}}
}

...Rect will then throw an exception on runtime, yet the test will still succeed.

Answer 1

To a certain extent, this is just part of the cost of doing business with dynamic languages. You get a lot of flexibility, otherwise known as "enough rope to hang yourself". Be careful with it.

To me, the problem suggest using different refactoring techniques than you would in a statically typed language. In a static language, you replace return types in part so that you can "lean on the compiler" to find what places may break. It is safe to do, and probably safer than modifying the return type in place.

In a dynamic language, you can't do that, so it is much safer to modify the return type in place, rather than replace it. Possibly, you modify it by adding your new class to it as a member, for the classes that need it.

Answer 2

If your code changes and your tests still pass, then there's either something wrong with your tests (i.e. you are missing an assertion), or the code didn't actually change.

What do I mean by that? Well, your tests describe the contracts between parts of your code. If the contract is "the return value must be iterable", then changing the return value from say an array to a list is not actually a change in contract, and therefore won't necessarily trigger a test failure.

In order to avoid missing assertions, you can use tools such as code coverage analysis (it won't tell you what parts of your code are tested, but it will tell you what parts definitely aren't tested), cyclomatic complexity and NPath complexity (which give you a lower bound on the number of assertions required to achieve full C1 and C2 code coverage) and mutation testers (which inject mutations in your code such as turning true to false, negative numbers into positive, objects into null etc. and check whether that makes tests fail).