1

We're looking to concentrate more on testing behaviour (as opposed to every class/method individually) in all our automated tests, as espoused by various high-profile online discussions recently. I like the idea of thinking about only what goes in and out of certain logical well-defined boundaries between components, but I'm having a hard time imagining how that relates to quite low-level detail of behaviour, that is only a small part of the overall requirement but still important to be correct.

As an example:

We have a component which accepts a customer order and takes some kind of significant action with it, let's say accepts it and starts processing manufacture. The main thrust of the requirement is about accepting and manufacturing, but there are many small things which need to be correct, and which if incorrect should cause an exception, for example that the order date needs to be valid. If we assume that I don't implicitly trust whatever date validation routine is being used, then there are loads of things I need to test there: basic format, leap years, not in the future, etc etc. In "non-behavioural" TDD I would definitely test those at the class/method level, and then never worry in my other tests about whether they'll work. But if I'm only testing behaviour, do I need to take all those success/fail cases and test them on the component boundary, for every situation where I'm interested in the date?

I can see a few options:

  • Massively tedious repetition of millions of tests, as on the low-level methods, but higher up.
  • As above, but condense "is the date valid" down into a separate method which I reuse in my behaviour tests, so although I do check that the date is valid, it's only one line per test. This is much more likely than the first option...
  • Test only broad behaviour higher-up, and do have a low level set of tests just to ensure the validation routines work. This seems sensible, but the importance of behaviour tests not relying on implementation details has been stressed - the actual validation routine being used is an implementation detail, invisible at the component boundary, so even if I test that routine, can I officially then not test it elsewhere as well somehow?

I'm leaning towards the middle option, but I'm quite new to this, and would be interested what you think about it!


Higher-level tests also exercise the interactions between components. Were your premise true, we could simply dispense with unit tests altogether and just use higher-level tests.

I think that's true (if taken to extremes), and there are people who would try to achieve exactly that. Understanding the reasons not to do that are partly the driver for my question.

I think potentially the difference between certain situations is to what extent these low-level behaviours are perceptible at the component boundary - in the case of date validation they are, I can feed lots of different values that I know should be valid or invalid in different ways into the interface and see differences in response; in other cases the unit in question does its work invisibly, at least from a behaviour perspective (I could test loads of stuff in a unit test, but perhaps the way it's being used in the larger component being tested doesn't give it much scope for variation).

Repetition vs Refactoring

The main reason I included the hyperbolic "millions of tests" option is another (unstated) principle that I also understand is good to aim for: "don't do too much in any one test, test one thing only". If you're also trying to follow that as much as possible, and think of the various validation failure cases as separate "things", it seems to conflict a bit with normal DRY/refactoring approaches - you can't squish the validation into one method and use it over and over, because then when it goes wrong you haven't got such specific feedback about what to fix (although you do at least have a valuable works/doesn't work response).

If this bothers you, follow the practice of TDD'ers everywhere, and only test public methods. Naturally, this doesn't preclude you making your "is the date valid" method a public method

I don't think I like the sounds of making something arbitrarily public (without changing anything else) just so I can test it - I'm happy enough changing my APIs when I need to change them, rather than so they fit my testing approach. Maybe bundling it off into another class (with public methods) if it might be useful, that makes more sense to me (and I've heard people suggest exactly that in TDD conversations). But the question is not really concerned with getting access to the low-level detail, it's about whether testing at that level at all is a good idea.

At the end of the day, common sense rules. If a piece of internal functionality is complex enough that you believe it needs to be covered by its own tests, then by all means, write tests for it. Conversely, I seldom write tests for trivial getter/setter methods, or methods that are prima facie correct.

No argument from me about that! Depsite what else I've written I'm sure we'll end up with a pragmatic mixture of approaches that we've chosen to give us the confidence in our code we need. This is helping me decide what that might be!

So far I'm erring towards accepting slightly coarser tests at the API level (for e.g. things like our date validation example), while also setting up comprehensive unit tests for certain underlying classes - that way you get the full behaviour coverage, and nothing can slip through the net at the level which truly matters, but also you can have good confidence in and ease of maintenance for the complex low-level units.

2 Answers 2

2

Massively tedious repetition of millions of tests, as on the low-level methods, but higher up.

That's a straw man... Naturally, you're testing the same functionality higher up, but at a different level of abstraction. Higher-level tests also exercise the interactions between components. Were your premise true, we could simply dispense with unit tests altogether and just use higher-level tests.

Condense "is the date valid" down into a separate method which I reuse in my behaviour tests, so although I do check that the date is valid, it's only one line per test.

That's called refactoring. It's a valid programming technique, and one that I would have assumed you already used here, in exactly the manner you described.

the actual validation routine being used is an implementation detail, blah blah.

If this bothers you, follow the practice of TDD'ers everywhere, and only test public methods. Naturally, this doesn't preclude you making your "is the date valid" method a public method, and testing it independently. It does have the virtue of making you think about your public API, and promoting code reuse by exposing those methods that are likely to be used more than once.

At the end of the day, common sense rules. If a piece of internal functionality is complex enough that you believe it needs to be covered by its own tests, then by all means, write tests for it. Conversely, I seldom write tests for trivial getter/setter methods, or methods that are prima facie correct.

2
  • Thanks a lot for this, I've waffled on a bit more in the question rather than max out the comments
    – frumious
    Sep 16, 2014 at 22:49
  • 1
    Food for thought: DRY doesn't really apply to unit tests, but only to the code under test. It's more important for tests to be self-contained than it is for them to be DRY. Sep 16, 2014 at 23:00
0

Something like a date validation function would be a genuinely reusable component, and so should;

  1. have tests written against it's api

  2. already exist, probably as part of your language standard library

  3. already be tested, otherwise use a different language/library

Then you just document somewhere 'we accept dates in (e.g.) ISO format'. Then for any kind of normal 'I want my website to work' testing, there are at most two possible test cases: legal date and non-legal date.

If you have some over-enthusiastic customer that has somehow specified their own date format, then it would become part of the behavioural tests. Try to find a better customer.

Of course, if you are certifying your code as suitable for use in the flight control system of a nuclear-powered passenger aeroplane, different levels of testing will apply. But that is well outside the scope in which any advice on cost-effective testing from people building consumer-facing websites is going to be useful.

1
  • Dates was just an example... What you say sort of makes sense, and of course any sufficiently complex chunk of functionality of that sort can be tested in relative isolation to make sure that it does what it says it will do, but I'm having trouble reconciling that with the "don't test implementation detail" rule for my behaviour tests. If all I'm testing is an interface, and that interface insists on certain facts about its inputs being true, but they go in as strings or whatever (rather than inherently valid self-contained objects), how do I verify that? Do I need to move my boundaries?
    – frumious
    Sep 16, 2014 at 13:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.