Is it a good idea to have separate test methods for every step?

Question

I am testing a REST api. Let's say it returns a JSON structure. What is the best approach to testing the server? Each test step can only succeed if all previous were successful.

Structure A: test everything at once

- Test method 1:
    - make server request
    - assert http response code was 200
    - assert returned file is not empty
    - assert returned file has valid JSON syntax
    - assert returned JSON contains key X

This seems to be the best solution.

Advantages:

• Only one server request
• I am testing behaviour as a whole "Does the server return a JSON with the key X?"

Structure B: gradually add asserts to each test

 - Test method 1:
     - make server request
     - assert http response code was 200
 - Test method 2:
     - make server request
     - assert returned file is not empty
 - Test method 3:
     - make server request
     - assert returned file has valid JSON syntax
 - Test method 4:
     - make server request
     - assert returned JSON contains key X

This is how I started out doing it and I was convinced that this should be the way to go because every method tests only one single thing and this creates better separation. But now I think, that since these are not unit tests my separation is not appropriate and I should test behavior as a whole.

Structure C: make request once and run separate test methods on cached response

- make server request and cache it (allow read-only access)

 - Test method 1:
     - assert http response code was 200 on cached server request
 - Test method 2:
     - assert returned file is not empty on cached server request
 - Test method 3:
     - assert returned file has valid JSON syntax on cached server request
 - Test method 4:
     - assert returned JSON contains key X on cached server request

Advantages:

• No repeated (expensive) server request
• Still has single-assert test methods

Which is the most sensible test structure to use?

Answer 1

Best practices always have a purpose, a reason behind them. It's always a good idea to consider these reasons in your design - especially when you are trying to decide how and how hard to follow these best practices.

In this case, the main reasoning behind making every test test a single thing is that if the first thing fails, the second one won't be tested. Since too many opinion makers seem to find merit in breaking everything to the smallest bits possible and wrapping every bit in as much bloat as possible, this gave birth to the idea that every test should contain a single assert.

Don't follow this blindly. Even if every test should test one thing, you should still put some thought into deciding how big or small each "thing" should be, and to do so you must keep in mind why you want every test to test one thing - to make sure a bug in the first thing is not leaving the second thing untested.

So, you need to ask yourself - "do I really need this guarantee here?"

Let's say there is a bug in the first test case - the HTTP response code is not 200. So you start hacking at the code, figure out why you did not get the response code you should have, and fix the problem. And now what?

• If you manually run the test again, to verify that your fix did solve the problem, you should run into any other problem hidden by the first failure.
• If you don't run it manually(maybe because it takes too long?), and just push your fix waiting for the automated tests server to run everything, then you may want to put different asserts in different tests. The cycles in this case are very long, so it's worthwhile to make the effort to discover as many bugs in each cycle.

There are few more things to consider:

Assertions dependencies

I know the tests you described are just an example, and your actual tests are probably more complicated - so what I'm going to say may not be valid with as much strength in the real tests, but it may still be somewhat effective so you may want to consider it.

If you have a REST service(or any other HTTP protocol) that returns responses in JSON format, you usually write a simple client class that lets you use the REST methods like regular methods that return regular objects. Assuming that client has separate tests to make sure it works, I would have ditched the first 3 asserts and keep only 4!

Why?

• The first assert is redundant - the client class should throw an exception if the HTTP response code is not 200.
• The second assert is redundant - if the response is empty, the result object will be null or some other representation of an empty object, and you won't have anywhere to put key X.
• The third assert is redundant - if the JSON is invalid, you'd get an exception when you try to parse it.

So you don't need to run all these tests - just run the fourth test, and if any of the bugs the first three tries to detect happen the test will fail with a proper exception before you even get the actual assert.

How do you want to receive the reports?

Let's say you don't get emails from a test server, but instead the QA department runs the tests and notifies you of failed tests.

Jack from QA knocks at your door. He says that the first test method failed, and the REST method returned a bad response code. You thank him, and start looking for the root cause.

Then comes Jen from QA, and say that the third test method failed - the REST method did not return a valid JSON in the response body. You tell her that you are already looking at that method, and you believe the same thing that caused it to return a bad exit code also caused it to return something that is not a valid JSON, and looks more like an exception stack trace.

You get back to working, but then Jim from QA arrives, saying that the fourth test method failed and there is no X key in the response...

You can't even look for the reason because it's hard to look at code when you don't have a computer screen. If Jim was quick enough he could have dodged in time...

Emails from the test server are easier to dismiss, but still - wouldn't you rather to just be notified ONCE that something is wrong with the test method, and look at the relevant test logs yourself?

Answer 2

If you can safely assume that making a server request with the same parameters will behave always the same, method B is almost pointless - why should you call four times the same method to get the same response data four times when one call is enough?

And if you cannot safely assume this and want to make it part of the test, you might be better off to run the test A multiple times.

The only hypothetical situation I see where B might have a benefit is when your testing framework allows only explicit test methods to be switched on and off, and you expect the necessity to do this for the individual steps of your test.

Alternative C seems to combine A with the one benefit I mentioned above for B. If your testing framework allows your code to be structured easily like that, without much overhead above B, this is a feasible approach. However, this adds some additional complexity to A, so I would only use it if I ever want to switch the individual tests on and off, otherwise apply the YAGNI principle and stick to the most simple solution (A).

TLDR: start with A if you are sure you always want all the asserts to be run in one test, refactor to C if you notice you need to have easier control from outside about the individual asserts.

Answer 3

Like any code, avoid premature optimisation. First write your tests so they are simple to read and simple to maintain. When the tests start to get too slow then optimise them. In your fairly simple example, A and B are both going to be easy to read and maintain so pick which ever one you want until things get too slow (structure B) or too complicated (structure A).

If your server is stateless then you could optimise by comparing the actual response with an expected response to check the whole of the message in one go. Obviously that will be at the expense of readability.

If your server is statefull and you need to make multiple slow api calls to get the server in a state for the test then you to take a different approach or your tests might take minutes to run. For example you could run a database update to inject data into a test database so you can quickly get an object in an appropriate state for testing. The test is quick and readable but more difficult to maintain. Alternatively you might be able to write a facade in front of the api so multiple api calls become a single api calls that more closely matches the business process you are testing.

Answer 4

Tests should not share things -- starting from scratch you avoid influence of one test on another one. This also makes you able to run tests in random order.
So the C way should not be accepted.

---

When writing any code (or maybe even creating anything else), always ask yourself: "why there is such practice?"
Why we say that there should be different tests for everything?

There are two cases when you need this:

1. when you can't rely on the "each test step can only succeed if all previous were successful"
2. when your tests don't have descriptive assert messages

There are two reasons why you face these cases:

1. "each test step can only succeed if all previous were successful" is really not applyable to your product feature
2. you don't have enough knowledge of the product because of lack of experience or time, or overwhelming product complexity

If you for some reason can't declare at least one of these reasons to have place just blindly take the structure B.

---

Otherwise (I hope you get here) you choose A.

---

Also you could ask this question on Software Quality Assurance & Testing Stackexchange site.