Is it OK to have multiple asserts in a single unit test?

Question

In the comment to this great post, Roy Osherove mentioned the OAPT project that is designed to run each assert in a single test.

The following is written on the project's home page:

Proper unit tests should fail for exactly one reason, that’s why you should be using one assert per unit test.

And, also, Roy wrote in comments:

My guideline is usually that you test one logical CONCEPT per test. you can have multiple asserts on the same object. they will usually be the same concept being tested.

I think that, there are some cases where multiple assertions are needed (e.g. Guard Assertion), but in general I try to avoid this. What is your opinion? Please provide a real world example where multiple asserts are really needed.

Answer 1

I don't think it's necessarily a bad thing, but I do think we should strive towards only having single asserts in our tests. This means you write a lot more tests and our tests would end up testing only one thing at a time.

Having said that, I would say maybe half of my tests actually only have one assert. I think it only becomes a code (test?) smell when you have about five or more asserts in your test.

How do you solve multiple asserts?

Answer 2

Tests should fail for one reason only, but that doesn't always mean that there should be only one Assert statement. IMHO it is more important to hold to the "Arrange, Act, Assert" pattern.

The key is that you have only one action, and then you inspect the results of that action using asserts. But it is "Arrange, Act, Assert, End of test". If you are tempted to continue testing by performing another action and more asserts afterwards, make that a separate test instead.

I am happy to see multiple assert statements that form parts of testing the same action. e.g.

[Test]
public void ValueIsInRange()
{
  int value = GetValueToTest();

  Assert.That(value, Is.GreaterThan(10), "value is too small");
  Assert.That(value, Is.LessThan(100), "value is too large");
} 

or

[Test]
public void ListContainsOneValue()
{
  var list = GetListOf(1);

  Assert.That(list, Is.Not.Null, "List is null");
  Assert.That(list.Count, Is.EqualTo(1), "Should have one item in list");
  Assert.That(list[0], Is.Not.Null, "Item is null");
} 

You could combine these into one assert, but that's a different thing from insisting that you should or must. There is no improvement from combining them.

e.g. The first one could be

Assert.IsTrue((10 < value) && (value < 100), "Value out of range"); 

But this is not better - the error message out of it is less specific, and it has no other advantages. I'm sure you can think of other examples where combining two or three (or more) asserts into one big boolean condition makes it harder to read, harder to alter and harder to work out why it failed. Why do this just for the sake of a rule?

NB: The code that I am writing here is C# with NUnit, but the principles will hold with other languages and frameworks. The syntax may be very similar too.

Answer 3

I have never thought that more than one assert was a bad thing.

I do it all the time:

public void ToPredicateTest()
{
    ResultField rf = new ResultField(ResultFieldType.Measurement, "name", 100);
    Predicate<ResultField> p = (new ConditionBuilder()).LessThanConst(400)
                                                       .Or()
                                                       .OpenParenthesis()
                                                       .GreaterThanConst(500)
                                                       .And()
                                                       .LessThanConst(1000)
                                                       .And().Not()
                                                       .EqualsConst(666)
                                                       .CloseParenthesis()
                                                       .ToPredicate();
    Assert.IsTrue(p(ResultField.FillResult(rf, 399)));
    Assert.IsTrue(p(ResultField.FillResult(rf, 567)));
    Assert.IsFalse(p(ResultField.FillResult(rf, 400)));
    Assert.IsFalse(p(ResultField.FillResult(rf, 666)));
    Assert.IsFalse(p(ResultField.FillResult(rf, 1001)));

    Predicate<ResultField> p2 = (new ConditionBuilder()).EqualsConst(true).ToPredicate();

    Assert.IsTrue(p2(new ResultField(ResultFieldType.Confirmation, "Is True", true)));
    Assert.IsFalse(p2(new ResultField(ResultFieldType.Confirmation, "Is False", false)));
}

Here I use multiple asserts to make sure complex conditions can be turned into the expected predicate.

I am only testing one unit (the ToPredicate method), but I am covering everything I can think of in the test.

Answer 4

When I'm using unit testing to validate high-level behavior, I absolutely put multiple assertions into a single test. Here's a test I'm actually using for some emergency notification code. The code that runs before the test puts the system into a state where if the main processor gets run, an alarm gets sent.

@Test
public void testAlarmSent() {
    assertAllUnitsAvailable();
    assertNewAlarmMessages(0);

    pulseMainProcessor();

    assertAllUnitsAlerting();
    assertAllNotificationsSent();
    assertAllNotificationsUnclosed();
    assertNewAlarmMessages(1);
}

It represents the conditions that need to exist at every step in the process in order for me to be confident that the code is behaving the way I expect. If a single assertion fails, I do not care that the remaining ones won't even get run; because the state of the system is no longer valid, those subsequent assertions wouldn't tell me anything valuable.* If assertAllUnitsAlerting() failed, then I wouldn't know what to make of assertAllNotificationSent()'s success OR failure until I determined what was causing the prior error and corrected it.

(* -- Okay, they might conceivably be useful in debugging the problem. But the most important information, that the test failed, has already been received.)

Answer 5

Another reason why I think, that multiple asserts in one method is not a bad thing is described in following code:

class Service {
    Result process();
}

class Result {
    Inner inner;
}

class Inner {
    int number;
}

In my test I simply want to test that service.process() returns the correct number in Inner class instances.

Instead of testing...

@Test
public void test() {
    Result res = service.process();
    if ( res != null && res.getInner() != null ) Assert.assertEquals( ..., res.getInner() );
}

I'm doing

@Test
public void test() {
    Result res = service.process();
    Assert.notNull(res);
    Assert.notNull(res.getInner());
    Assert.assertEquals( ..., res.getInner() );
}

Answer 6

I think there are plenty of cases where writing multiple asserts is valid within the rule that a test should only fail for one reason.

For example, imagine a function that parses a date string:

function testParseValidDateYMD() {
    var date = Date.parse("2016-01-02");

    Assert.That(date.Year).Equals(2016);
    Assert.That(date.Month).Equals(1);
    Assert.That(date.Day).Equals(0);
}

If the test fails it is because of one reason, the parsing is incorrect. If you would argue that this test can fail for three different reasons, you would IMHO be too fine grained in your definition of "one reason".

Answer 7

I don't know of any situation where it would be a good idea to have multiple assertions inside the [Test] method itself. The main reason people like to have multiple Assertions is they are trying to have a one [TestFixture] class for each class being tested. Instead, you can break up your tests into more [TestFixture] classes. This allows you to see multiple ways in which the code may not have reacted in the way you expected, instead of just the one where the first assertion failed. The way you achieve this is you have at least one directory per class being tested with lots of [TestFixture] classes inside. Each [TestFixture] class would be named after the specific state of an object you will be testing. The [SetUp] method will get the object into the state described by the class name. Then you have multiple [Test] methods each asserting different things you would expect to be true, given the object's current state. Each [Test] method is named after the thing it is asserting, except perhaps it might be named after the concept instead of just an English readout of the code. Then each [Test] method implementation only needs a single line of code where it is asserting something. Another advantage of this approach is it makes the tests very readable as it becomes quite clear what you are testing, and what you expect just by looking at the class and method names. This will also scale better as you begin to realize all the little edge cases you want to test and as you find bugs.

Usually this means that the final line of code inside the [SetUp] method should store a property value or return value in a private instance variable of the [TestFixture]. Then you might assert multiple different things about this instance variable from different [Test] methods. You also might make assertions about what different properties of the object under test are set to now that it is in the desired state.

Sometimes you need to make assertions along the way as you are getting the object under test into the desired state in order to make sure you did not mess up before getting the object into the desired state. In that case, those extra assertions should appear inside the [SetUp] method. If something goes wrong inside the [SetUp] method, it will be clear that something was wrong with the test before the object ever got into the desired state you intended to test.

One other problem you may run into is you may be testing an Exception that you expected to be thrown. This can tempt you into not following the above model. However, it can still be achieved by catching the exception inside the [SetUp] method and storing it into an instance variable. This will allow you to assert different things about the exception, each in it's own [Test] method. You can then also assert other things about the object under test to make sure there were no unintended side effects from the exception being thrown.

Example (this would be broken up into multiple files):

namespace Tests.AcctTests
{
    [TestFixture]
    public class no_events
    {
        private Acct _acct;

        [SetUp]
        public void SetUp() {
            _acct = new Acct();
        }

        [Test]
        public void balance_0() {
            Assert.That(_acct.Balance, Is.EqualTo(0m));
        }
    }

    [TestFixture]
    public class try_withdraw_0
    {
        private Acct _acct;
        private List<string> _problems;

        [SetUp]
        public void SetUp() {
            _acct = new Acct();
            Assert.That(_acct.Balance, Is.EqualTo(0));
            _problems = _acct.Withdraw(0m);
        }

        [Test]
        public void has_problem() {
            Assert.That(_problems, Is.EquivalentTo(new string[] { "Withdraw amount must be greater than zero." }));
        }

        [Test]
        public void balance_not_changed() {
            Assert.That(_acct.Balance, Is.EqualTo(0m));
        }
    }

    [TestFixture]
    public class try_withdraw_negative
    {
        private Acct _acct;
        private List<string> _problems;

        [SetUp]
        public void SetUp() {
            _acct = new Acct();
            Assert.That(_acct.Balance, Is.EqualTo(0));
            _problems = _acct.Withdraw(-0.01m);
        }

        [Test]
        public void has_problem() {
            Assert.That(_problems, Is.EquivalentTo(new string[] { "Withdraw amount must be greater than zero." }));
        }

        [Test]
        public void balance_not_changed() {
            Assert.That(_acct.Balance, Is.EqualTo(0m));
        }
    }
}

Answer 8

If your test fails, you won't know whether the following assertions will break, too. Often, that means you'll be missing valuable information to figure out the source of the problem. My solution is to use one assert but with several values:

String actual = "val1="+val1+"\nval2="+val2;
assertEquals(
    "val1=5\n" +
    "val2=hello"
    , actual
);

That allows me to see all failed assertions at once. I use several lines because most IDEs will display string differences in a compare dialog side-by-side.

Answer 9

Having multiple assertions in the same test is only a problem when the test fails. Then you might have to debug the test or analyse the exception to find out which assertion it is that fails. With one assertion in each test it's usually easier to pinpoint what's wrong.

I can't think of a scenario where multiple assertions are really needed, as you can always rewrite them as multiple conditions in the same assertion. It may however be preferrable if you for example have several steps to verify the intermediate data between steps rather than risking that the later steps crash because of bad input.

Answer 10

The goal of the unit test is to give you as much information as possible about what is failing but also to help accurately pinpoint the most fundamental problems first. When you know logically that one assertion will fail given that another assertion fails or in other words there is a dependency relationship between the test then it makes sense to roll these as multiple asserts within a single test. This has the benefit of not littering the test results with obvious failures which could have been eliminated if we bailed out on the first assertion within a single test. In the case where this relationship does not exist the preference would naturally be then to separate these assertions into individual tests because otherwise finding these failures would require multiple iterations of test runs to work out all of the issues.

If you then also design the units/classes in such a way that overly complex tests would need to be written it makes less burden during testing and probably promotes a better design.

Answer 11

If you have multiple asserts in a single test function, I expect them to be directly relevant to the test you are conducting. For example,

@Test
test_Is_Date_segments_correct {

   // It is okay if you have multiple asserts checking dd, mm, yyyy, hh, mm, ss, etc. 
   // But you would not have any assert statement checking if it is string or number,
   // that is a different test and may be with multiple or single assert statement.
}

Having a lot of tests (even when you feel that it is probably an overkill) is not a bad thing. You can argue that having the vital and most essential tests is more important. SO, when you are asserting, make sure your assert statements are correctly placed rather than worrying about multiple asserts too much. If you need more than one, use more than one.

Answer 12

Yes, it is ok to have multiple assertions as long as a failing test gives you enough information to be able to diagnose the failure. This is going to depend on what you're testing and what the failure modes are.

Proper unit tests should fail for exactly one reason, that’s why you should be using one assert per unit test.

I've never found such formulations to be helpful (that a class should have one reason to change is an example of just such an unhelpful adage). Consider an assertion that two strings are equal, this is semantically equivalent to asserting that the length of the two strings are the same and each character at the corresponding index is equal.

We could generalize and say that any system of multiple assertions could be rewritten as a single assertion, and any single assertion could be decomposed into a set of smaller assertions.

So, just focus on the clarity of the code and the clarity of the test results, and let that guide the number of assertions you use rather than vice versa.

Answer 13

This question is related to the classic problem of balancing between spaghetti and lasagna code issues.

Having multiple asserts could easily get in the spaghetti problem where you do not have an idea what the test is about, but having single assert per test could make your testing equally unreadable having multiple tests in a big lasagna making the finding which test does what impossible.

There are some exceptions, but in this case keeping the pendulum in the middle is the answer.

Answer 14

The answer is very simple - if you test a function that changes more than one attribute, of the same object, or even two different objects, and the correctness of the function depends on the results of all of those changes, then you want to assert that every one of those changes has been properly performed!

I get the idea of a logical concept, but the reverse conclusion would say that no function must ever change more than one object. But that's impossible to implement in all cases, in my experience.

Take the logical concept of a bank transaction - withdrawing an amount from one bank account in most cases HAS TO include adding that amount to another account. You NEVER want to separate those two things, they form an atomic unit. You might want to make two functions (withdraw/addMoney) and thus write two different unit tests - in addition. But those two actions have to take place within one transaction and you also want to make sure that the transaction works. In that case it's simply not enough to make sure the individual steps were successful. You have to check both bank accounts, in your test.

There might be more complex examples that you wouldn't test in a unit test, in the first place, but instead in an integration or acceptance test. But those boundaries are fluent, IMHO! It's not that easy to decide, it's a matter of circumstances and maybe personal preference. Withdrawing money from one and adding it to another account is still a very simple function and definitely a candidate for unit testing.

Answer 15

I don't even agree with the "fail for one reason only" in general. What is more important is that tests are short and reads clearly imo.

This is not always achievable though and when a test is complicated a (long) descriptive name and testing fewer things makes more sense.