I've inherited an application that has a suite of tests that drive me batty. But one of the design decisions that this test suite takes that completely leaves me scratching my head is the separation of tests and their expectations into separate files.

To explain, there are functional tests that have test methods with decorators. The tests themselves do the setup, run the test procedure, and any teardown if necessary. But the expectations for the tests are defined in separate JSON or YAML files. The decorators on the test methods do some magic to find the expectations file, and the appropriate data set to match the test method.

This pattern is also used for input values and fixtures. I find this overly designed, but I can maybe understand input values and fixtures being external.

This drives me nuts because if I have to make a logic change, I have to search through multiple files to find the expected values and change them. I've wasted days trying to fix the test suite after any major refactoring is made.

I just want to know if there is any valid reason for this design pattern. I can't think of one. If someone has suggestions or information on why this pattern was chosen or possibly prescribed, please add links to appropriate articles, docs, and opinions.

--- EDIT ---

Sorry to have left this question hanging.

I realize I didn't give a lot of background. So let me give you more. I've heard arguments that there are situations where non-developers (ie project managers, accountants, etc...) might need to edit tests or expectations. This is not the case. These tests are only to be written, read, and executed by developers and/or automation. I also heard that it could be the case with BDD, where you might use simple nouns to be placeholders for expected values. This isn't the case either, we don't have any BDDs, and whether or not they are useful is irrelevant to this question. As stated, they are supposed to be functional tests. There are unit tests, but those are more like integration tests, they test a full path of execution.

The project's language is Python and the tests are nose tests. The application is a web service API. No browser front end, it is designed to interact mostly with our client SDK's (mobile and various device platforms). Our customers are primarily developers of mobile games.

Here is an example of how the tests are laid out:


The JSON files in expected_values and input_values each contain a single JSON object that contains an object for each test.

Here's an example of an input_values file. These files contain test data representing request data that would be posted to the API from the SDKs. The first object default represents the version of the API that is being tested. This I think is unnecessary because we don't have very many versions, and the ones that are, are drastically different.

  "_register": {
    "default": {
      "json_data": {
        "device": {
          "identifier": "dev_1",
          "name": "testuser device 1"
  // ...

Then the expected_values files contain the values a test expects to be returned in the API response. Each file is one object that contains an object for each test.

  "_register": {
    "default": {
      "created_at": "regex:DATETIME",
      "display_name": null,
      "email": null,
      "is_confirmed": false,
      "state": "anonymous",
      "uuid": "regex:UUID"
  // ...

And here is what that test would look like.

""" imports and such """

class AnonymousUserTestCase(base.BaseFunctionalTest):
    def setUp(self):
        super(AnonymousUserTestCase, self).setUp()
        self.resp = None
        self.session_id = None
        self.email = None

    """ OTHER TESTS """
    def _register(self, json_data=None):
        """Test register user."""
        path = self.populate_api_version('/{version}/auth/register')
        resp = self.app_post_json(path, data=json_data)
        self.email = resp.parsed_data.get('username')
        self.uuid = resp.parsed_data.get('uuid')
        return reps
    """ MORE TESTS """

Still to this day I have to live with these tests. And because of them, testing has been problematic. We've taken a different approach to testing this app. It's more of a Blackbox test. We have a dev environment set up, and a Simulated device then makes request calls to the API and checks for the expected responses. This makes it easier for the Client SDK developers because they can see and control how the API is used and if it meets their expectations. But it does cause some longer turnaround time because it requires the services developers to make the API changes, deploy it and then run the automated testers. Usually, by that time, the Client SDK developers have already written the tests.

We are only now performing minor updates and hotfixes to this API and hope to replace it completely with a new project.

  • 3
    I see some reasonable argument why it could have happened. Some people writing the tests are not those who write the input/expected data or some specific people that are not familiar with code wanted to be able to change the input/ouput for more thorough test. The developer that though about it wanted to separate the concern and though it was a nice idea when he made it or he has that idea because the data were often changing. Note : I'am not saying that this is a good idea, just what could made it happened.
    – Walfrat
    Commented Nov 21, 2017 at 10:33
  • I did something similar with a C++ project of mine so I wouldn't have to recompile every time I added a test; I also felt that it was easier to look at the test expectations if they were in a separate, simpler format. Not sure if this was a good idea, though, or if it merits turning into an answer.
    – DylanSp
    Commented Nov 21, 2017 at 14:31
  • @salvobeta do the expectation files recap the context and subject of the test, or do they just contain the expectations - i.e. you systematically have to look at two files to have a complete picture of what's going on? Can you provide code samples? Commented Nov 22, 2017 at 14:28

5 Answers 5


From your description, there is no plausible reason for keeping actions and expected results that separate.

Exporting data to external resource files makes sense if there is a high probability that the data will have to change often without the processing code changing. For instance, this can make the difference between having to perform an expensive redeployment and a simple edit + maybe a restart.

But this concerns data that are reassembled by the computer. A test suite is processed by both computers and humans, and it is important that the test suite maintainer can easily correlate conditions and outcomes to understand what is happening. Separating them with no tool support for reassembling them seems a very bad decision. I can't think of any compensating benefit to justify this. It's not like the expected results for a test will are expected to change very often.

  • To support your second paragraph, I've worked on a project where ruleset were mostly dynamically added by the end user, and thus individual customers were allowed to write their own input/expectations for certain high level integration tests so they could effectively test their own ruleset (done via JSON files that were overwritten/added during the release (not build!) pipelines). Technically, that wasn't testing our code but rather the behavior of their ruleset, but the success of these tests was essential to get a green light for updating the production code.
    – Flater
    Commented Jul 6, 2020 at 10:27

Writing test code in a very real sense is like writing application code in that organization and reuse of code is best practice. However, I tend to agree with you that I want to be able to read a test method and know exactly what it's doing. The only exception to this is the setup and teardown, and only if the setup and teardown are doing tasks that will be potentially common to all tests without interfering with the actual test itself (setup and closing of the database connection for instance).

So by that logic, I tend to create clear and concise helper classes with static methods that organize the tasks that need to be performed for a test in very few calls. In other words, rather than have to hunt down the test file to understand what is being tested, I can see clearly code like:

User john = TestUserHelper.createUser("JOHN SMITH");

// The test

john = TestUserHelper.loadUser("JOHN SMITH");

assertEquals(ADMIN, john.getPermissions());

Clearly there is a lot going on underneath, but it is straightforward to read and it is easy to change. Most of the work being done is performed in TestUserHelper. I think this is something to strive for frankly. And yes, since writing test code is very much like writing application code, this is something to strive for in application code as well. Hiding complexity sometimes only serves to hide it, not to reduce it.


Yes there could be a valid reason. Not saying this is the case here, just that it is possible.

Separating the expectations into declarative file formats separate from the test code would allow a non-developer (like a businesses analyst) to write tests of business logic. This could be useful for acceptance tests.

  • Not quite. This would apply if inputs and expectations were kept separate from the test logic implementation (as appears to be the case here) but are kept together with each other in order to form a complete test case from the business analysts point of view. This is not the case here. Commented Nov 21, 2017 at 15:59

Yes there is: BDD.

BDD proponents believe this is fine:

First the product owner and tester produce a simple scenario file describing behavior and expectations.

Given a data fixture When a page is created with 20 items on a page Then expected lines on a page is 4 with data items in a line 5

Next developers write a harness and flesh out the steps required to meet the behavior.

Check the file history for the author, it'll likely be a (former) product owner.

You could gang with your team and change all of the "behavior" style tests to a different style of integration test at one time.


For Fine Grain Detection of Failure in Integration/End to End tests

The harnessed system is undergoing a number of concurrent/sequential long running operations. The complexity is sufficiently high that littering expectations throughout easily obscures the control flow, and because the test takes so long to run it makes sense to collect as many observations as possible to identify issues (much like a compiler compiling past the first syntax error).

The expectations themselves are needing to be kept separate from the harnessed system, as it is irksome to have to discard an entire run at the 60min mark because something didn't look right.

The expectations are also separated from each other so that when something does go sproing, what is wrong can be more easily identified (by naming the expectation/expectation set) well.

And yes, these tests designed poorly are not fun to maintain.

They also don't fit well in most modern Unit Testing frameworks, so custom code runners makes for even more headache.

The Inputs/Expectations are not Trivial

Such as:

  • consuming a large transfer file through an input process.
  • comparing a 2mb report byte by byte.

It makes sense to have these as a file external to the test fixture, otherwise you would have 20K lines of base64 text per test. Not fun.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.