What are the drawbacks of modeling (unit) test fixtures in JSON instead of using ObjectFactories?

Question

When writing java tests for an application, be they unit tests or testing a broader scope, the java community tends to model fixtures in terms of object factories that produce fixtures of a defined state.

eg:

public class TestFixtureGeneratorA{
    static public A modelAState1(){
        A a = new A();
        a.setFoo(1);
        a.setBar("bar");
    }
    static public A modelAState2(){
        A a = new A();
        a.setFoo(2);
        a.setBar("baz");
    }
}

Whereas, coming from a BDD or Agile methodologies perspective, it would seem more sensible to encode fixtures in terms of their applied scenario and feature or test.

eg:

//file: src/test/resources/fixtures/scenarioA/interaction1/A.json
{
    foo: 1,
    bar: "bar"
}

//file: src/test/resources/fixtures/scenarioA/interaction2/A.json
{
    foo: 2,
    bar: "baz"
}

//file: src/test/resources/fixtures/scenarioA/interaction2/ids.json
[42, 101, 1337]

And then have your test execute in the scope provided by scenarioA interactionX.

I have seen this kind of thing in agile driven teams that model from business requirement to code (ie: they maintain a full BDD stack), none of them java shops) and once in a relatively big industry application. But trying to look up literature on the practice was fruitless. There does seem to be one implementation at Corballis' GitHub repository providing annotation based test infrastructure, similar to the one described above but other than that the Web seems devoid of work on this subject.

Given the succinctness of JSON and the probably better formalization/integration aspects of the JSON based fixture mechanic, how come it didn't get discussed until now? Are there any apparent drawbacks I'm missing?

Note: This is taken from personal experience in a multitude of projects, mostly in Java, highly diversified (some where millions SLOC, others have insane performance requirements, etc).

Answer 1

I've written tests that were JSON supported before. Here were the drawbacks:

• JSON is not compiler-checked. This means if you add a field to your models, you won't know your test JSON is broken until you try to run the tests and they all fail.
• JSON is more difficult to produce than code. In my case our models were quite complex, so I would start by copying another JSON file and tweaking it.
• JSON can't contain comments. You can't document the important features of the JSON file that make it appropriate for the scenario.
• IDE integrations for JSON are not as advanced as those for code. For example I can jump to definitions and attributes in code, but I can't do that for JSON. There is a separation between the code and the JSON files that may make it difficult understand at a glance which JSON file is being used in a test. Some knowledge of a naming convention (in your example) would be necessary, and I would have to navigate to the file in my IDE or OS, and parse it visually and with text search.

it would seem more sensible to encode fixtures in terms of their applied scenario and feature or test.

What does this have to do with using JSON or not? You can produce an object tailored to your feature under test in code. That would be my preference. It is a trade-off because the code will likely be quite long.

You can treat long construction code by creating builders for complex models. So instead of

A a = new A();
a.setFoo(2);
a.setBar("baz");

You could write

A a = new ABuilder().foo(2).bar("baz").build();

Or create a factory

A a = new AFactory(2, "baz").create();

Answer 2

If an integrationtest (or JSON based test) fails you know that something is wrong but you cannot be shure which component caused the problem. It could be the JSON parser/mapper, the businesslogic or any other module involved in the test.

If a unittest (in isolation with factorymethods) fails it is likely that the single module under test caused the problem.

For me the info that "something is wrong" is good enough so i prefer using deserialised testdata (from JSON, csv or xml)

This question is an issue between unittesting versus integrationtesting where both have their pros and cons.