The problem occurs while doing TDD. After a couple of test pass, the return types of some class/module change. In a statically typed programming language, if a previous mocked object was used in the tests of some other class and was not modified to reflect the type change, then compilation errors will occur.

For dynamic languages however, the change in return types might not be detected and the tests of the other class will still pass. Sure there might be integration tests that should fail later on, but unit tests would erroneously pass. Is there any way how to avoid this?

Updating with a trivial sample (on some made up language)...

Version 1:

Calc = {
    doMultiply(x, y) {return x * y}
//.... more code ....

// On some faraway remote code on a different file
Rect = {
    computeArea(l, w) {return Calc.doMultipy(x*y)}

// test for Rect
testComputeArea() { 
    Calc = new Mock()
    Calc.expect(doMultiply, 2, 30) // where 2 is the arity
    assertEqual(30, computeArea)

Now, on version 2:

// I change the return types. I also update the tests for Calc
Calc = {
    doMultiply(x, y) {return {result: (x * y), success:true}}

...Rect will then throw an exception on runtime, yet the test will still succeed.

  • 1
    What the answers so far seem to miss is that the question is not about the tests involving the changed class X, but the tests of class Y which depends on X and thus gets tested against a different contract than what it runs against in production. – Bart van Ingen Schenau Jan 2 '13 at 17:11
  • I just asked this question on SO, myself, in regard to Dependency Injection. See Reason 1: A dependent class can be changed at run time (think testing). We both have the same mindset but are lacking great explanations. – Scott Coates Jan 4 '13 at 23:53
  • I'm re-reading your question but am getting a little confused on the interpretation. Can you provide an example? – Scott Coates Jan 5 '13 at 0:16

To a certain extent, this is just part of the cost of doing business with dynamic languages. You get a lot of flexibility, otherwise known as "enough rope to hang yourself". Be careful with it.

To me, the problem suggest using different refactoring techniques than you would in a statically typed language. In a static language, you replace return types in part so that you can "lean on the compiler" to find what places may break. It is safe to do, and probably safer than modifying the return type in place.

In a dynamic language, you can't do that, so it is much safer to modify the return type in place, rather than replace it. Possibly, you modify it by adding your new class to it as a member, for the classes that need it.

| improve this answer | |

If your code changes and your tests still pass, then there's either something wrong with your tests (i.e. you are missing an assertion), or the code didn't actually change.

What do I mean by that? Well, your tests describe the contracts between parts of your code. If the contract is "the return value must be iterable", then changing the return value from say an array to a list is not actually a change in contract, and therefore won't necessarily trigger a test failure.

In order to avoid missing assertions, you can use tools such as code coverage analysis (it won't tell you what parts of your code are tested, but it will tell you what parts definitely aren't tested), cyclomatic complexity and NPath complexity (which give you a lower bound on the number of assertions required to achieve full C1 and C2 code coverage) and mutation testers (which inject mutations in your code such as turning true to false, negative numbers into positive, objects into null etc. and check whether that makes tests fail).

| improve this answer | |
  • +1 Either something wrong with tests or code didn't actually change. This took me a while to get used to after years of C++/Java. The contract between parts in dynamic languages of code should not be WHAT is returned, but what that thing that is returned contains and what it can do. – MrFox Jan 11 '13 at 15:44
  • @suslik: Actually, this is not about static vs. dynamic languages, but rather about Data Abstraction using Abstract Data Types vs. Object-Oriented Data Abstraction. The ability for one object to simulate another object as long as it behaves indistinguishable (even if they are of completely different types and instances of completely different classes) is fundamental to the whole idea of OO. Or put another way: if two objects behave the same, they are of the same type, regardless of what their classes say. Put yet another way: classes aren't types. Unfortunately, Java and C# get this wrong. – Jörg W Mittag Jan 12 '13 at 1:34
  • Assuming that the mocked unit is 100% covered and there's no assertion missing: how about a more subtle change like "should return null for foo" to "should return empty string for foo". If someone changes that unit and its test some consuming units may break and due to the mock this is transparent. (And no: at time of writing or using the mocked module, per "contract", it's not necessary to handle empty strings as return value because it's supposed to return non-empty strings or null only ;) ). (Only proper integration testing of both modules in interaction could catch this.) – try-catch-finally Mar 11 '18 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.