Suppose we have a service that returns users in certain categories, e.g.

  • Listens to a certain artist.
  • Listens to a certain playlist.
  • Listens to a certain genre.
  • Is between ages X and Y.
  • Is located in geographic area Z.

Suppose an external CategoryService manages valid categories, and we use a CategoryServiceFixture for our integration testing. This fixture may or may not come with categories defined, e.g. we may use a fixture preloaded with actual production categories, or we may reset the fixture and add our own categories, e.g.

categoryServiceFixture.addArtist("Red Hot Chili Peppers");
categoryServiceFixture.addPlaylist("Techno Bunker");
categoryServiceFixture.addAgeGroup("90s Teens", 1970, 1990);
categoryServiceFixture.addCity("New York");

Again, our service, ListenerTargetingService, fetches users who match a certain filter, e.g.

    new ListenerTargetingFilterBuilder()
        .addGenres("House", "Tech House")

I don't think it's necessary to write tests for every permutation of dimensions. I think it's sufficient to write tests such as...

@Test test_findByArtist_noneFound();
@Test test_findByArtist_someFound();
@Test test_findByArtist_invalidArtist();
@Test test_findByArtists_allFound();
@Test test_findByArtists_invalidArtist();

...for each dimension, and then a set of tests that combines filters:

@Test test_findByAllCategories_exactlyOneFound();
@Test test_findByAllButOneCategory_exactlyNFound();
@Test test_findByAllButTwoCategories_exactlyNFound();
// Doesn't need to be exhaustive. This is probably enough.

So here's where I got to thinking: Okay, so I can programmatically generate seed data so that the number of permutations found in each of the combined cases is always computable.

I can go ahead on my own, but it seems there would already exist a well-defined pattern or idiom for this kind of testing. What is it? Searching things like "permutation testing" mostly leads to results about methods in statistics, or papers on "orthogonal testing methods" (which are on target but too academic).

In case a package or tool can be recommended, I'm specifically using Java, JUnit, Mockito, Dagger, etc.

  • Hmm, conceptually, if you're testing to see if exactly one artist is found, would you be testing something different if that particular artist were Eminem or another artist? Same can be said for the other criteria. If you don't have reason to think that a particular data would be treated differently from another, swapping out that data for something else equally valid is not improving your unit tests in any way. In fact, you might argue that the variation in data is worse, as it makes your tests complicated and no longer predictable, so that you just have to trust that the test worked.. – Neil Nov 20 '18 at 13:07

I think the thing you are looking for is data driven testing.

When you have this kind of situation where there are lots of similar tests that all follow the same pattern data driven testing, where you write a single test and list a whole bunch of different parameters to pass to it, can cut down the volume of code you need to write without compromising test coverage.

So in your case you could write a test which takes a filter and an expected result set as parameters.

Then have a whole block of various filters and the expected results all of which run the same test.

  • Or perhaps Property-Based Testing. – Robert Harvey Nov 20 '18 at 17:41
  • You were both right. With the help of JUnit 5's @TestFactory I'm implementing these patterns. Thanks. – Andrew Cheong May 7 at 6:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.