0

In answer to Is testable code better code? I showed how time-dependent code could be testable by using mocks rather than modifying the implementation:

def time_of_day():
    return datetime.datetime.utcnow().strftime('%H:%M:%S')

Test code:

@unittest.patch('datetime.datetime.utcnow')
def test_handle_leap_second(self, utcnow_mock):
    utcnow_mock.return_value = datetime.datetime(
        year=2015, month=6, day=30, hour=23, minute=59, second=60)
    actual = time_of_day()
    expected = '23:59:60'
    self.assertEquals(actual, expected)

The problem here (apart from Python not handling leap seconds) is that the test code assumes that utcnow will return a datetime.datetime object in absolutely all situations. That may be true right now for this specific function with the current interpreter etc., but if the return type changes for whatever reason the test will be passing when it probably should be failing. So for a robust test suite it would be better to guarantee that the mock return value has the same class as the original function return value. This is obviously only possible in languages where the return type is enforced by the compiler or interpreter. You could work around this by modifying a return value of the original function rather than creating your own:

utcnow_mock.return_value = datetime.datetime.utcnow().replace(
    year=2015, month=6, day=30, hour=23, minute=59, second=60)

Now we're operating with a different set of assumptions:

  • replace modifies the object returned by utcnow rather than returning some different object. This will not be an issue if the relevant object properties are writable.
  • utcnow always returns objects with the same class.

Which method is more likely to produce a robust test suite? Please consider that unlike this example creating and then modifying an object may be simpler than creating it with the correct properties from scratch.

1

It depends on what you mean by "robust test suite".

One possible definition would be "my tests never test the wrong thing, even if the underlying requirements of the system change", and I think this is the definition you're leaning towards with your problem description. Honestly, I think this is an unsolvable problem (what if in the next revision of the library, utcnow() doesn't return any value at all and instead is intended to be used to acquire a lock on a DB table?). Also note that when you demand "don't give false positive" (don't have a test fail if there's actually no bug), you usually have to tolerate some false negatives (the test might say there's no bug, but actually there is one).

Another possible definition would be "my test always detect when a change would trigger a bug in my program, in terms of how the requirements were understood when this test was written". I think this is the more commonly used definition (although most of us don't explicitly think about these definitions nor the distinction between them). Under this definition, you would tend to prefer using mock objects rather than mutate existing objects -- but if it turns out that it's more convenient to mutate existing objects than create mock ones, that's okay too. The key insight here though is that if you're worried that utcnow() might change in such a way as to introduce a bug in your system, then you shouldn't write tests only to verify that time_of_day() is performing as you expect, but you should additionally write tests that verify utcnow() is performing as you expect.

Note that under this philosophy, we're aiming for "no false negatives" (if there's a bug, it should be the case that some test somewhere will fail), and so we tolerate a bit of false positive (sometimes a test will say there's a bug, but after inspection, we realize there's actually no bug, the test was just testing the wrong thing).

  • +1 I hadn't thought of it that way - you're right, adding smoke tests for library code would be more useful than jumping through hoops when testing my own code. – l0b0 Jul 4 '15 at 20:11
  • This technique is particularly useful when you discover a bug in the library code, and you write a workaround, but the workaround will fail if the library ever fixes their bug. You write a test asserting that the bug exists in the library (and thus your workaround is needed). Then, if the library ever fixes the bug, you'll immediately be alerted via a failing test. I usually add a comment in the test reminding me where to look for the workaround that now has to be removed. – Nebu Pookins Jul 4 '15 at 20:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.