Should tests perform a single assertion, or are multiple related assertions acceptable

Question

Assume a client is making a request to an API endpoint that returns a JSON response where the structure and data change depending on whether the request was successful or not. Possible responses may look as follows:

Scenario with a success response

{
  "status": XXX,
  "data": [{
    ...
  }]
}

Scenario with a failure response

{
  "status": XXX,
  "errors": [{
    ...
  }]
}

Example of test scenarios for the above would be:

• Assert the expected status code
• Assert the expected JSON structure
• Assert the expected JSON data

When performing tests where you have more than one assertion you can perform, is it recommended to provide a single test for each assertion, or group the assertions into a single test?

Answer 1

Ask yourself what unit tests are good for. The main purpose is that after changing your code, you run all the unit tests, and if they all pass, you have a bit more confidence that your code is fine. For this purpose, what matters is the number if independent assertions that you passed, so for this purpose a single unit test with 100 assertions is no problem.

The second purpose is that if your unit test fails, you know as precisely as possible what is wrong, saving you time searching for the exact failure. One assert per unit test saves you some time locating the exact failure.

Depending on your environment, this will help more or less. So it’s a judgement call how important this is for you in your particular environment. Some environments only report the first assertions in a unit test, others report all. That obviously makes a difference.

But most important is test coverage. If you are a good unit tester writing one assertion per unit test, but after 50 tests you get bored and stop writing tests, and I write unit-tests with 20 assertions each and after 20 unit tests I still keep going, then I’m doing better.

PS. Some people take the rule “one assert per test” and use it to justify doing stupid things. Let’s say my code produced five values, and my unit test knows what the five results should be. Now you can go against the rule:

Assert(result1 == expected1)
  ...
Assert(result5 == expected5)

Or you "follow the rules" and write

Assert(result1 == expected1 && ... && result5 == expected5)

And guess what: The second leaves me with not a clue what the actual problem is. The first one will even in the worst case tell me the first problem.

Answer 2

Each unit test should assert a single requirement.

Now, you may not always have formal requirements for every method. Methods are written in pursuit of fulfilling such formal requirements. But given that a method should do only one thing, should have a single purpose, you should already have a fair idea of what to expect from the method in terms of behavior.

Consider your first example. You could always just assert against the entire output (pseudocode follows):

Assert.AreEqual(result, @"{
  'status': XXX,
  'data': [{
    ...
  }]
}");

But such tests are brittle, and when they break, there is no indication of what went wrong or what requirement is not being met. You have to look through the output and see where the actual problem is, and then analyze the problem to determine what requirement is not being met.

Asserting on a single requirement means you can give your unit test a meaningful name that states specifically what requirement is being tested. There are a number of ways to do that.

Given_UserIsAuthenticated_When_InvalidAccountNumberIsUsedToWithdrawMoney_Then_TransactionWillFail()  

When the test fails, it will show up in your test runner with the meaningful name and you will know immediately which requirement is not being met.

Answer 3

I think it's worth keeping in mind that, if you are running tests as part of your development/design loop, it doesn't matter very much how to describe a test failure. You don't need a lot of information emitted by the test when you know the root cause is the line of code you just changed.

Where it tends to matter is when a bunch of new code gets merged in as a block, and we need the detailed signals to track down the problem(s) that were just introduced.

When performing tests where you have more than one assertion you can perform, is it recommended to provide a single test for each assertion, or group the assertions into a single test?

So let's assume for the start that we're talking specifically about tests that follow the Arrange-Act-Assert recipe. So we create a system, and then we send that system zero of more messages, and finally we want to measure whether the system now satisfies some constraint.

That constraint usually has the form of "extract some value from the system, and compare it to some expected value".

Such a comparison might have N different checks to be applied.

One option is to have a different test for each check; the tests will duplicate the arrange and act phases, but will have slightly different assert phases. The good news is that all of the checks are run, and don't interfere with each other. If the test is properly isolated, the tests can be run concurrently. The bad news is that, without additional structure, your test harness won't necessarily understand that the tests should be bundled together, and you'll need to look in several places to get a complete understanding of what's going on when things fail.

Another option is to put all of the checks into a single test. The good news is that any given scenario is in one place, so the code describes the specification you are trying to satisfy in one place, rather than leaving it scattered everywhere. The bad news is that a single failed check trips the assert; the subsequently scheduled checks are never evaluated, leaving us with an incomplete picture of what's going on.

A third option is to treat each check as a predicate, AND the results, and assert on the outcome. That gives you a single test with all of the logic, and all of the checks being evaluated, but it gives you only a weak signal about which check was the problem.

A fourth option is similar - collect the results of the check together, but assert on the results, rather than simple boolean predicates. Here's a ordered list of expected outcomes, here's an ordered list of actual outcomes, assert if they don't match. This gives you a lot of everything -- all of the checks are in one place AND all of the checks run AND you get clear messages about all of the failures.

The bad news is that this fourth approach isn't how SUnit was designed, and therefore isn't how JUnit was designed, and therefore isn't how all of the descendants of those tools were designed.

Functional programmers might be more comfortable with this last approach; we create a list of checks, and then map to create a list of check results, and then fold those check results together.

Note that none of these approaches does particularly well if the test happens to trip a precondition in the arrange or act phase.

Best Practice? Your choice

1) Document what you know, document what you guess, document your decision, document the outcomes you expect, and review that document from time to time, re-evaluating the call.

2) Experiment, accumulate evidence, and improve your ability to guess which external factors are meaningful in a given context.

Answer 4

I know my reply is a bit late, but I hope it helps. In my experience, it's generally recommended to have a single assertion per test case. This helps to keep the tests focused and specific, which makes it easier to identify and fix issues when they arise. However, if you have multiple related assertions, it's okay to group them into a single test case as long as they’re all testing the same functionality. Also, you can do functional testing using automation tools to find gaps, bugs, or any issues in software performance. It's a critical part of the testing process and can help you catch issues that you might have missed with unit testing alone. Ultimately, it's up to you and your team to decide what testing strategy works best for your particular situation.