Should I create mock objects from scratch or modify the result of real calls in weakly typed languages?

Question

In answer to Is testable code better code? I showed how time-dependent code could be testable by using mocks rather than modifying the implementation:

def time_of_day():
    return datetime.datetime.utcnow().strftime('%H:%M:%S')

Test code:

@unittest.patch('datetime.datetime.utcnow')
def test_handle_leap_second(self, utcnow_mock):
    utcnow_mock.return_value = datetime.datetime(
        year=2015, month=6, day=30, hour=23, minute=59, second=60)
    actual = time_of_day()
    expected = '23:59:60'
    self.assertEquals(actual, expected)

The problem here (apart from Python not handling leap seconds) is that the test code assumes that utcnow will return a datetime.datetime object in absolutely all situations. That may be true right now for this specific function with the current interpreter etc., but if the return type changes for whatever reason the test will be passing when it probably should be failing. So for a robust test suite it would be better to guarantee that the mock return value has the same class as the original function return value. This is obviously only possible in languages where the return type is enforced by the compiler or interpreter. You could work around this by modifying a return value of the original function rather than creating your own:

utcnow_mock.return_value = datetime.datetime.utcnow().replace(
    year=2015, month=6, day=30, hour=23, minute=59, second=60)

Now we're operating with a different set of assumptions:

• replace modifies the object returned by utcnow rather than returning some different object. This will not be an issue if the relevant object properties are writable.
• utcnow always returns objects with the same class.

Which method is more likely to produce a robust test suite? Please consider that unlike this example creating and then modifying an object may be simpler than creating it with the correct properties from scratch.

Answer 1

It depends on what you mean by "robust test suite".

One possible definition would be "my tests never test the wrong thing, even if the underlying requirements of the system change", and I think this is the definition you're leaning towards with your problem description. Honestly, I think this is an unsolvable problem (what if in the next revision of the library, utcnow() doesn't return any value at all and instead is intended to be used to acquire a lock on a DB table?). Also note that when you demand "don't give false positive" (don't have a test fail if there's actually no bug), you usually have to tolerate some false negatives (the test might say there's no bug, but actually there is one).

Another possible definition would be "my test always detect when a change would trigger a bug in my program, in terms of how the requirements were understood when this test was written". I think this is the more commonly used definition (although most of us don't explicitly think about these definitions nor the distinction between them). Under this definition, you would tend to prefer using mock objects rather than mutate existing objects -- but if it turns out that it's more convenient to mutate existing objects than create mock ones, that's okay too. The key insight here though is that if you're worried that utcnow() might change in such a way as to introduce a bug in your system, then you shouldn't write tests only to verify that time_of_day() is performing as you expect, but you should additionally write tests that verify utcnow() is performing as you expect.

Note that under this philosophy, we're aiming for "no false negatives" (if there's a bug, it should be the case that some test somewhere will fail), and so we tolerate a bit of false positive (sometimes a test will say there's a bug, but after inspection, we realize there's actually no bug, the test was just testing the wrong thing).