I've inherited an application that has a suite of tests that drive me batty. But one of the design decisions that this test suite takes that completely leaves me scratching my head is the separation of tests and their expectations into separate files.

To explain, there are functional tests that have test methods with decorators. The tests themselves do the setup, run the test procedure, and any teardown if necessary. But the expectations for the tests are defined in separate JSON or YAML files. The decorators on the test methods do some magic to find the expectations file, and the appropriate data set to match the test method.

This pattern is also used for input values and fixtures. I find this overly designed, but I can maybe understand input values and fixtures being external.

This drives me nuts because if I have to make a logic change, I have to search through multiple files to find the expected values and change them. I've wasted days trying to fix the test suite after any major refactor is made.

I just want to know if there is any valid reason for this design pattern. I can't think of one. If someone has suggestions or information on why this pattern was chosen or possibly prescribed, please add links to appropriate articles, docs, and opinions.

| improve this question | | | | |
  • 2
    I see some reasonable argument why it could have happened. Some people writing the tests are not those who write the input/expected data or some specific people that are not familiar with code wanted to be able to change the input/ouput for more thorough test. The developer that though about it wanted to separate the concern and though it was a nice idea when he made it or he has that idea because the data were often changing. Note : I'am not saying that this is a good idea, just what could made it happened. – Walfrat Nov 21 '17 at 10:33
  • I did something similar with a C++ project of mine so I wouldn't have to recompile every time I added a test; I also felt that it was easier to look at the test expectations if they were in a separate, simpler format. Not sure if this was a good idea, though, or if it merits turning into an answer. – DylanSp Nov 21 '17 at 14:31
  • @salvobeta do the expectation files recap the context and subject of the test, or do they just contain the expectations - i.e. you systematically have to look at two files to have a complete picture of what's going on? Can you provide code samples? – guillaume31 Nov 22 '17 at 14:28

From your description, there is no plausible reason for keeping actions and expected results that separate.

Exporting data to external resource files makes sense if there is a high probability that the data will have to change often without the processing code changing. For instance, this can make the difference between having to perform an expensive redeployment and a simple edit + maybe a restart.

But this concerns data that are reassembled by the computer. A test suite is processed by both computers and humans, and it is important that the test suite maintainer can easily correlate conditions and outcomes to understand what is happening. Separating them with no tool support for reassembling them seems a very bad decision. I can't think of any compensating benefit to justify this. It's not like the expected results for a test will are expected to change very often.

| improve this answer | | | | |

Yes there could be a valid reason. Not saying this is the case here, just that it is possible.

Separating the expectations into declarative file formats separate from the test code would allow a non-developer (like a businesses analyst) to write tests of business logic. This could be useful for acceptance tests.

| improve this answer | | | | |
  • Not quite. This would apply if inputs and expectations were kept separate from the test logic implementation (as appears to be the case here) but are kept together with each other in order to form a complete test case from the business analysts point of view. This is not the case here. – Lutz Prechelt Nov 21 '17 at 15:59

Yes there is: BDD.

BDD proponents believe this is fine:

First the product owner and tester produce a simple scenario file describing behavior and expectations.

Given a data fixture When a page is created with 20 items on a page Then expected lines on a page is 4 with data items in a line 5

Next developers write a harness and flesh out the steps required to meet the behavior.

Check the file history for the author, it'll likely be a (former) product owner.

You could gang with your team and change all of the "behavior" style tests to a different style of integration test at one time.

| improve this answer | | | | |

Writing test code in a very real sense is like writing application code in that organization and reuse of code is best practice. However, I tend to agree with you that I want to be able to read a test method and know exactly what it's doing. The only exception to this is the setup and teardown, and only if the setup and teardown are doing tasks that will be potentially common to all tests without interfering with the actual test itself (setup and closing of the database connection for instance).

So by that logic, I tend to create clear and concise helper classes with static methods that organize the tasks that need to be performed for a test in very few calls. In other words, rather than have to hunt down the test file to understand what is being tested, I can see clearly code like:

User john = TestUserHelper.createUser("JOHN SMITH");

// The test

john = TestUserHelper.loadUser("JOHN SMITH");

assertEquals(ADMIN, john.getPermissions());

Clearly there is a lot going on underneath, but it is straightforward to read and it is easy to change. Most of the work being done is performed in TestUserHelper. I think this is something to strive for frankly. And yes, since writing test code is very much like writing application code, this is something to strive for in application code as well. Hiding complexity sometimes only serves to hide it, not to reduce it.

| improve this answer | | | | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.