The phase "unit test" began as a description of testing individual functions and
other small bits of code / capabilities. If that is all you are testing--bits of
your system, in isolation from one another--it is no wonder that you are losing
confidence in your tests to be a fair measure of overall code quality or "does
it do what it's supposed to do?"
Very narrow testing helps a little, but just a little. It's great that you
aren't getting production failures that trace back to things that have been
tested. The problem seems to be the white space in between--the untested
execution paths, the combination of elements that happens when all of the parts
come together that are not currently tested. Without being too Pollyannaish,
there is your problem. Or a good part of it, at any rate.
This is perhaps heresy, but it's worked well for me: Unit tests are no longer
about isolated, narrow units. They are a general purpose automated testing
facility that can and should test not only individual functions, classes, and
objects, but combinations of them, at varying levels of composition.
I may be running
py.test or whatever the local "unit test"
framework is, but I eagerly add tests up to and including tests of the entire
integrated application or service. I run complex examples of the entire app
functionality, and do so not just against simple data cases, but also against
some gnarly, ugly, complex edge-cases.
It isn't as easy to test that way. It requires creating some
mock objects, and/or building entire
virtual machines and populating them with servers, data sets, and other glue
logic just to run the tests. It's more work to set up that test scenario, but
with virtualization and cloud instances, tools like
Fabric and Vagrant make
it doable in ways that it just wasn't, five or so years back.
It can even be complicated to determine if a test succeeded or failed. If you're
used to testing small functions, it's hard to see why--but start trying to judge
whether you've properly manipulated a complex data structure and it becomes more
clear. It can be even worse if you're trying to see a particular path through a
GUI or web app to ensure it "did what it's supposed to do." But with modern
tools, it is possible--and the results are extremely helpful at exercising the
Testing can never be a proof of correctness under all possible conditions. But
exercising your app or service as a whole, with complex and edge/corner case
examples designed to stress your logic...first it will show you where a lot of
your bugs lie. ("The truth will set you free — but first it will make you
But as you fix those and your system is executing hard cases successfully and
routinely, then your confidence in its correctness and robustness will grow.
As you find new bugs and fix them, you will add tests for them, and other cases
like them you hadn't previously considered. On subsequent passes at the testing
job, you'll find new things you want to test, or new variants you want to test
against. You add them in, and next run, you're testing even more of the total
If anyone wants to argue that I'm not really unit testing--that I'm doing some
form of integration, module, or system testing--well whatever! I use the same
(unit testing) frameworks, test runners, "did it pass?" reporting that I would
for testing a 2-line function. If you want to give fancier names for some of the
tests, knock yourself out. What I care about is that the tests are automated,
that I can run them easily for every software build, and that they test as much
of the total functionality as possible.
Long story short, instead of losing faith in testing, double down on it. Do the
testing that you need, but aren't currently doing. Test things together, in ever
more challenging and complete combinations, against ever more execution
environments. You'll be amazed at the bugs that shake out, and at how much you
learn about your system interactions as a result. Untimately, you'll be amazed
at how you move with confidence from version to version, because your shakedown
has been real and systematic.