Is there a valid reason to separate expectations from tests?

Question

I've inherited an application that has a suite of tests that drive me batty. But one of the design decisions that this test suite takes that completely leaves me scratching my head is the separation of tests and their expectations into separate files.

To explain, there are functional tests that have test methods with decorators. The tests themselves do the setup, run the test procedure, and any teardown if necessary. But the expectations for the tests are defined in separate JSON or YAML files. The decorators on the test methods do some magic to find the expectations file, and the appropriate data set to match the test method.

This pattern is also used for input values and fixtures. I find this overly designed, but I can maybe understand input values and fixtures being external.

This drives me nuts because if I have to make a logic change, I have to search through multiple files to find the expected values and change them. I've wasted days trying to fix the test suite after any major refactoring is made.

I just want to know if there is any valid reason for this design pattern. I can't think of one. If someone has suggestions or information on why this pattern was chosen or possibly prescribed, please add links to appropriate articles, docs, and opinions.

--- EDIT ---

Sorry to have left this question hanging.

I realize I didn't give a lot of background. So let me give you more. I've heard arguments that there are situations where non-developers (ie project managers, accountants, etc...) might need to edit tests or expectations. This is not the case. These tests are only to be written, read, and executed by developers and/or automation. I also heard that it could be the case with BDD, where you might use simple nouns to be placeholders for expected values. This isn't the case either, we don't have any BDDs, and whether or not they are useful is irrelevant to this question. As stated, they are supposed to be functional tests. There are unit tests, but those are more like integration tests, they test a full path of execution.

The project's language is Python and the tests are nose tests. The application is a web service API. No browser front end, it is designed to interact mostly with our client SDK's (mobile and various device platforms). Our customers are primarily developers of mobile games.

Here is an example of how the tests are laid out:

project/
  tests/
    functional/
      api/
        expected_values/
          anonymous_user_test_case.json
          ...
        input_values/
          anonymous_user_test_case.json
          ...
        anonymous_user_scenario_test.py

The JSON files in expected_values and input_values each contain a single JSON object that contains an object for each test.

Here's an example of an input_values file. These files contain test data representing request data that would be posted to the API from the SDKs. The first object default represents the version of the API that is being tested. This I think is unnecessary because we don't have very many versions, and the ones that are, are drastically different.

{
  "_register": {
    "default": {
      "json_data": {
        "device": {
          "identifier": "dev_1",
          "name": "testuser device 1"
        }
      }
    }
  }
  // ...
}

Then the expected_values files contain the values a test expects to be returned in the API response. Each file is one object that contains an object for each test.

{
  "_register": {
    "default": {
      "created_at": "regex:DATETIME",
      "display_name": null,
      "email": null,
      "is_confirmed": false,
      "state": "anonymous",
      "uuid": "regex:UUID"
    }
  },
  // ...
}

And here is what that test would look like.

""" imports and such """

class AnonymousUserTestCase(base.BaseFunctionalTest):
    def setUp(self):
        super(AnonymousUserTestCase, self).setUp()
        self.resp = None
        self.session_id = None
        self.email = None

    """ OTHER TESTS """
    @check_expected
    @with_input_data
    def _register(self, json_data=None):
        """Test register user."""
        path = self.populate_api_version('/{version}/auth/register')
        resp = self.app_post_json(path, data=json_data)
        self.email = resp.parsed_data.get('username')
        self.uuid = resp.parsed_data.get('uuid')
        return reps
    
    """ MORE TESTS """

Still to this day I have to live with these tests. And because of them, testing has been problematic. We've taken a different approach to testing this app. It's more of a Blackbox test. We have a dev environment set up, and a Simulated device then makes request calls to the API and checks for the expected responses. This makes it easier for the Client SDK developers because they can see and control how the API is used and if it meets their expectations. But it does cause some longer turnaround time because it requires the services developers to make the API changes, deploy it and then run the automated testers. Usually, by that time, the Client SDK developers have already written the tests.

We are only now performing minor updates and hotfixes to this API and hope to replace it completely with a new project.

Answer 1

From your description, there is no plausible reason for keeping actions and expected results that separate.

Exporting data to external resource files makes sense if there is a high probability that the data will have to change often without the processing code changing. For instance, this can make the difference between having to perform an expensive redeployment and a simple edit + maybe a restart.

But this concerns data that are reassembled by the computer. A test suite is processed by both computers and humans, and it is important that the test suite maintainer can easily correlate conditions and outcomes to understand what is happening. Separating them with no tool support for reassembling them seems a very bad decision. I can't think of any compensating benefit to justify this. It's not like the expected results for a test will are expected to change very often.

Answer 2

Writing test code in a very real sense is like writing application code in that organization and reuse of code is best practice. However, I tend to agree with you that I want to be able to read a test method and know exactly what it's doing. The only exception to this is the setup and teardown, and only if the setup and teardown are doing tasks that will be potentially common to all tests without interfering with the actual test itself (setup and closing of the database connection for instance).

So by that logic, I tend to create clear and concise helper classes with static methods that organize the tasks that need to be performed for a test in very few calls. In other words, rather than have to hunt down the test file to understand what is being tested, I can see clearly code like:

User john = TestUserHelper.createUser("JOHN SMITH");

// The test
john.changePermissions(ADMIN);

john = TestUserHelper.loadUser("JOHN SMITH");

assertEquals(ADMIN, john.getPermissions());

Clearly there is a lot going on underneath, but it is straightforward to read and it is easy to change. Most of the work being done is performed in TestUserHelper. I think this is something to strive for frankly. And yes, since writing test code is very much like writing application code, this is something to strive for in application code as well. Hiding complexity sometimes only serves to hide it, not to reduce it.

Answer 3

Yes there could be a valid reason. Not saying this is the case here, just that it is possible.

Separating the expectations into declarative file formats separate from the test code would allow a non-developer (like a businesses analyst) to write tests of business logic. This could be useful for acceptance tests.

Answer 4

Yes there is: BDD.

BDD proponents believe this is fine:

First the product owner and tester produce a simple scenario file describing behavior and expectations.

Given a data fixture When a page is created with 20 items on a page Then expected lines on a page is 4 with data items in a line 5

Next developers write a harness and flesh out the steps required to meet the behavior.

Check the file history for the author, it'll likely be a (former) product owner.

You could gang with your team and change all of the "behavior" style tests to a different style of integration test at one time.

Answer 5

For Fine Grain Detection of Failure in Integration/End to End tests

The harnessed system is undergoing a number of concurrent/sequential long running operations. The complexity is sufficiently high that littering expectations throughout easily obscures the control flow, and because the test takes so long to run it makes sense to collect as many observations as possible to identify issues (much like a compiler compiling past the first syntax error).

The expectations themselves are needing to be kept separate from the harnessed system, as it is irksome to have to discard an entire run at the 60min mark because something didn't look right.

The expectations are also separated from each other so that when something does go sproing, what is wrong can be more easily identified (by naming the expectation/expectation set) well.

And yes, these tests designed poorly are not fun to maintain.

They also don't fit well in most modern Unit Testing frameworks, so custom code runners makes for even more headache.

The Inputs/Expectations are not Trivial

Such as:

• consuming a large transfer file through an input process.
• comparing a 2mb report byte by byte.

It makes sense to have these as a file external to the test fixture, otherwise you would have 20K lines of base64 text per test. Not fun.