How to write tests in TDD for downloading and unpacking a file?

Question

So I want to write this function that downloads a file and unpacks it in python using TDD.

The function will look like this approximately

import urllib.request
import tarfile

def download_and_unpack(tarball_url, destination)
    datastream = urllib.request.urlopen(tarball_url)
    thetarfile = tarfile.open(fileobj=datastream, mode="r|gz")
    thetarfile.extractall(destination)

Now, I don't want to test the implementation but the behaviour, and I don't need to test the python standard library, so I don't know where to start.

I was thinking of creating a tarball in /tmp, pass it to the function and see that the files are unpacked, but that feels like testing too much at once. Any pointers or advice on how to start here?

As someone suggested, I could wrap each line in a custom interface and test the function by mocking these instead, but then I would have to test the interfaces themselves, and I feel I would end up in the same dead end...

Answer 1

In reality, I would probably not test this function in more than one TDD step - chaining three ready-made system functions together isn't particular error-prone, and if it fails, finding the root cause should be done pretty quick with one type of test alone.

However, lets do this for the sake of an exercise. This function consists mainly of two steps (as it's name says), so it should be possible to develop it in two TDD cycles. One possible approach here could be introducing a testmode parameter, which is set to "False" as default, but when set to True, it will write the downloaded data stream directly into a file.

This gives you the opportunity to first write a test to verify the download works, and make the first implementation of the function look like this:

def download_and_unpack(tarball_url, destination, testmode=False)
    datastream = urllib.request.urlopen(tarball_url)
    if testmode:
         # add code here to write datastream to a file

You need to call download_and_unpack with testmode=True in this test, of course.

Afterwards, you write a test asserting the function does both steps (download and unpack), then complete it. Finally, you add a test verifying when testmode=True, the result will be the same with no intermediate file.

Of course, I don't think in this contrived example it would be really worth the hassle, bot for larger functions with more complex code sections, allowing access to some intermediate steps' data, and logging it or writing it to some temporary file is an old, simple and practice-proven technique which can play well together with TDD.