16

Recently at work we've been having some differences of opinion with regard to Parameterized testing. Normally we use a TDD-style (or at least try to) so I understand the benefits of that approac. However, I'm struggling to see the gain parameterized tests bring. For reference, we work on a service and it's libraries which are exposed via a RESTful interface.

What I've seen so far is tests that are, at least using JUnit within Eclipse:

  • Lacking in detail - when a test fails it's very hard to see the parameters which caused it to fail
  • Often complicated to create
  • Tend to be created after the code has been written - strictly not a drawback as such but do people set out with parameterized tests in mind when they start a piece of code?

If anyone has any examples of where they are really useful or even any good hints for using them that would be fantastic. I want to make sure I'm not just being obstinate because I personally don't choose to use them and see whether they are something we should consider being part of our testing arsenal.

Improve this question
2
  • 1
    The problem is not with the idea but with the clunky library. In C# the syntax is way friendlier, when you use say MbUnit. Yes, it is a good idea. Add your own code to make this process easier - read stuff from files - whatever works. Also look at how MsTest handles this.
    – Job
    Jul 10, 2011 at 14:07
  • We (Square) wrote Burst to address some of these issues with Parameterized. It generally adds less boilerplate, and makes it pretty clear where a test failed.
    – Daniel Lubarov
    Oct 21, 2014 at 18:55

6 Answers 6

Reset to default
5

The problem with testing any piece of software is that complexity blows up pretty quickly. The fact is, you can't test all possible combinations of parameters passed to your methods. Phadke advocates a Design of Experiments (DOE) approach, which allows generation of the likely list of parameter values that need to be tested.

The idea is that, even though you are not testing exhaustively, most defects cause a "fault region" to occur rather than an isolated point fault. The DOE approach Phadke advocates using orthogonal arrays samples the parameter space finely enough to hit all possible fault regions.

Isolated faults will probably not be identified, but these are generally fewer than fault regions.

The DOE approach gives you a systematic way of choosing the parameter values to vary.

Improve this answer
0
4

They can be useful for ensuring that your code handles not just the happy path, but also the edge cases. After you know your code works with normal variables, parametrize the test case and make sure nulls and 0's, empty strings, large numbers, long strings, weird Unicode characters, etc., also work fine.

Improve this answer
4

There are at least two flavors of parameterized tests, at least in JUnit 4.8. Those are: Parameterized Tests (@RunWith(Parameterized.class)) which requires a data source, which generates/reads predefined parameter configurations, and Theories (@RunWith(Theories.class)) which, given one or more set of possible inputs per argument type can exercise specification of given methods. It looks more-less like this:

  • specify some possible values (@DataPoints) for string arguments (like null, empty string, non-empty string, really long string)
  • specify some possible values (@DataPoints) for Animal class arguments (like null, Dog instance, Cat instance, Bird instance)
  • prepare @Theory which accepts a String parameter and an Animal parameter. it will get executed with every possible combination of the possible parameter values (in given example that'd be 4x4 = 16 combinations, including (null, null))
  • if method under test cannot accept some combination, use Assume.assumeThat static imports to filter out invalid combinations (eg. when you want to check method's behavior for non-empty strings, one of the first lines would be "assume that is not null"

As written before - it doesn't make sense to test every possible combination of every method (it explodes test sets, imagine testing a method with 5 parameters, each one having just 5 possible values: 5**5 -> over 3000 test runs!), but for mission-critical methods (like API methods) I'd encourage it, just to be on the safe side...

Improve this answer
2

Parameterized tests work well for testing functions/features that have simple input when you want to test a variety of input.

They don't work well for testing different functionality and complex input. They should not be used as a convenience structure to write less code.

Improve this answer
2
  • 1
    Why should parameters not be used as a convenience to write less code? There is no great virtue in providing an exhaustive listing of a large(ish) set of test cases.
    – Jonathan Eunice
    Feb 12, 2015 at 23:21
  • 1
    How does your answer provide any more information than the other 5 answers?
    – Adam Zuckerman
    Feb 13, 2015 at 3:19
1

Generic example:

  • Methods with string arguments. Use parametrized tests to test different inputs and their expected outputs. Is much more practical to have a list of pairs (input, expected) than to write a TC for each pair.

  • Apply same scenario on different arguments. We have a scenario that works with the Animal object and the have a lot of subclasses such as Dog, Cat, Bird. Create a list of the available animals and test the scenario on them.

Concrete for webservice:

  • From the string arguments example above. Test what happens with different arguments of the same type but different values.
Improve this answer
-2

One case where I use lots of parameterized tests in a TDD-ish manner is writing parsers -- I can start with a list if input and expected output and then write the code so it passes all test cases.

But I've seen a few horrors of parameterized testing. No, Virginia, your test suite should not need it's own unit tests.

Improve this answer
1
  • 1
    Ideally parametised tests should be of the form "does item(n) in the actual output match item(n) in the expect output" or similar, and in that case no testing needed. But for anything more complex I'd prefer to seen a clean parametised test or two with their own test cases than the usual handwaving "my (test) code is obviously correct". If that was true you wouldn't be writing test cases at all. Obviously it's possible to go overboard with the complexity and I'm not arguing that there's no line, but I think there are cases where testing tests is a good idea.
    – Мסž
    Mar 28, 2011 at 22:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.