1

When writing java tests for an application, be they unit tests or testing a broader scope, the java community tends to model fixtures in terms of object factories that produce fixtures of a defined state.

eg:

public class TestFixtureGeneratorA{
    static public A modelAState1(){
        A a = new A();
        a.setFoo(1);
        a.setBar("bar");
    }
    static public A modelAState2(){
        A a = new A();
        a.setFoo(2);
        a.setBar("baz");
    }
}

Whereas, coming from a BDD or Agile methodologies perspective, it would seem more sensible to encode fixtures in terms of their applied scenario and feature or test.

eg:

//file: src/test/resources/fixtures/scenarioA/interaction1/A.json
{
    foo: 1,
    bar: "bar"
}

//file: src/test/resources/fixtures/scenarioA/interaction2/A.json
{
    foo: 2,
    bar: "baz"
}

//file: src/test/resources/fixtures/scenarioA/interaction2/ids.json
[42, 101, 1337]

And then have your test execute in the scope provided by scenarioA interactionX.

I have seen this kind of thing in agile driven teams that model from business requirement to code (ie: they maintain a full BDD stack), none of them java shops) and once in a relatively big industry application. But trying to look up literature on the practice was fruitless. There does seem to be one implementation at Corballis' GitHub repository providing annotation based test infrastructure, similar to the one described above but other than that the Web seems devoid of work on this subject.

Given the succinctness of JSON and the probably better formalization/integration aspects of the JSON based fixture mechanic, how come it didn't get discussed until now? Are there any apparent drawbacks I'm missing?

Note: This is taken from personal experience in a multitude of projects, mostly in Java, highly diversified (some where millions SLOC, others have insane performance requirements, etc).

Improve this question
4
  • 2
    Maybe I'm not understanding your intent, but this seems far less flexible and only saves you from writing code that would trivial to write while requiring (potentially in a library) moderately complex code to support it. If you have a test for a bug that fails when given a 1,000,000 element array, would you rather have a 1MB test file, or a couple lines of code? What about things that don't have an easily serializable (to JSON) representation, e.g. a cyclic graph or a network socket?
    – Derek Elkins left SE
    Aug 16 '17 at 11:24
  • Fair point about large fixtures, but I think at that point You are testing a set of fixtures distributed over an array, not an array as the fixture. Regarding unserializable fixtures. Well... The whole point of formal testing is based on observation of reproducible environments and actions. So I would say Your fixture isn't the socket, it is the calls to java.net.Socket, which is an interface so inherently You can describe it in JSON. May also have a different fixture for the Outputstream it produces.
    – Kiriakos Krastillis
    Aug 16 '17 at 11:46
  • Regarding the code inclusion, I think that stuff is quite easily produced as a lib and libs we as a community tend to depend on a lot :-) I'm not sure though if the Corballis implementation does everything right. I think there is room for improvement. In case of spinning Your own, I tested around last night and a combination of PathMatchingResourcePatternResolver and MockRestServiceServer does take You a long way. It is a hack though.
    – Kiriakos Krastillis
    Aug 16 '17 at 11:55
  • I believe that you are struggling with the issue of domain specific languages for test case construction. While JSON has some benefits as far as ease of writing, as a DSL it is a very poor substitute. I notice that you have directories and files and file contents, all of which seem to be independent of each other. A good DSL helps to organise your complexity, not increase your complexity across multiple domains (file systems, files, data, etc.).
    – BobDalgleish
    Aug 16 '17 at 22:43
5

I've written tests that were JSON supported before. Here were the drawbacks:

  • JSON is not compiler-checked. This means if you add a field to your models, you won't know your test JSON is broken until you try to run the tests and they all fail.
  • JSON is more difficult to produce than code. In my case our models were quite complex, so I would start by copying another JSON file and tweaking it.
  • JSON can't contain comments. You can't document the important features of the JSON file that make it appropriate for the scenario.
  • IDE integrations for JSON are not as advanced as those for code. For example I can jump to definitions and attributes in code, but I can't do that for JSON. There is a separation between the code and the JSON files that may make it difficult understand at a glance which JSON file is being used in a test. Some knowledge of a naming convention (in your example) would be necessary, and I would have to navigate to the file in my IDE or OS, and parse it visually and with text search.

it would seem more sensible to encode fixtures in terms of their applied scenario and feature or test.

What does this have to do with using JSON or not? You can produce an object tailored to your feature under test in code. That would be my preference. It is a trade-off because the code will likely be quite long.

You can treat long construction code by creating builders for complex models. So instead of

A a = new A();
a.setFoo(2);
a.setBar("baz");

You could write

A a = new ABuilder().foo(2).bar("baz").build();

Or create a factory

A a = new AFactory(2, "baz").create();
Improve this answer
4
  • Additionally you have extra access to the file system to load the JSON data. But UnitTests should not use resources outside the JVMs RAM.
    – Timothy Truckle
    Aug 16 '17 at 21:33
  • Thanks @Samuel, good point about the IDE Integration aspect. Do You think refactoring is an issue? I think, given a good modularization of the code it should be manageable. I'm kind of partial about having too many classes facilitating object creation though, especially if it is driven by testing.
    – Kiriakos Krastillis
    Aug 18 '17 at 13:53
  • @Timothy Truckle, where would You put the line regarding resources? A java class created to produce fixtures is also only a file. I totally agree that, especially regarding unit testing, one should have minimal dependencies. But from a pragmatic standpoint having a lib deserialize JSON is not that much different from a lib that Mocks another class.
    – Kiriakos Krastillis
    Aug 18 '17 at 13:54
  • @KiriakosKrastillis Class files are red by the JVM once per invocation and with lots of effort for efficiency under the hood. Test data in recource file are read when ever you access them, most likely for each test method in setup. - I personally configure my test data at the top of my test class. I choose my design so keep that configuration part as small as possible...
    – Timothy Truckle
    Aug 18 '17 at 14:00
1

If an integrationtest (or JSON based test) fails you know that something is wrong but you cannot be shure which component caused the problem. It could be the JSON parser/mapper, the businesslogic or any other module involved in the test.

If a unittest (in isolation with factorymethods) fails it is likely that the single module under test caused the problem.

For me the info that "something is wrong" is good enough so i prefer using deserialised testdata (from JSON, csv or xml)

This question is an issue between unittesting versus integrationtesting where both have their pros and cons.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.