3

I'm practicing test driven development while implementing a network simulator: devices communicate with each other and implement little bits of functionality that respond to messages. Each device has its own view of the network topology and can be individually powered on or off.

Timing is important. If the Device is in the simulation, has powered, and is powered off, it should be powered on after exactly two update calls.

Below is an extract of a test case I have written, trying to decompose the problem. Be warned that to a large degree I'm making the design up as I fly, so it could be the case that I haven't decomposed the objects involved enough.

let device = Device()
device.hasPower = true
device.isPoweredOn = false

// There is hidden state that we do not know about. 
// Does this make it a bad test?
// Is this instead some kind of integration test?
device.update()
XCTAssertEqual(device.log.last, .poweringOn)
device.update()
XCTAssertEqual(device.log.last, .poweredOn)
XCTAssertEqual(device.isPoweredOn, true)

Notice how device depends upon its previous state between the two assert statements.

Taking a step back, what is actually going on is I'm verifying that the state of device is equal to a sequence of state changes:

  • .poweringOn, then
  • .poweredOn.

The question is: how can I write effective tests when the number of elements in the sequence becomes large? (or, equivalently, when the simulation depends upon time, or equivalently, when you have an update loop).

Research

Why not look at what other people do for testing simulation software. OMNeT++ I'm looking at you.. I'm looking for an approach that doesn't involve setting up external tests. That is, I'd like to tests specific bits of functionality in my codebase because it is convenient with today's tooling and I believe it is more rigourous.

I looked at other questions on this site but couldn't find anything related to testing simulations. I searched for "depends on previous state" and "simulation testing" but couldn't find anything that matched what I was looking for. I also tried googling "multiple assets unit testing", and "simulation testing", but again the results are not relevant to my problem.

3
  • "There is hidden state that we do not know about." - so, in TDD, tests are supposed to be "stand-ins" for actual client code (production code that will use, or interact with, your Device class), which then allows you to get a feel for how the interaction between the two will work. If this state is hidden, then client code has no way of noticing any difference between the initial state, and the state after the first update call, and so neither should the test. You have a hidden side-effect here, and (from client perspective) surprising behavior after the second update call. Jun 18 at 15:30
  • So, maybe the test itself is more of a symptom, and the primary thing to look at is, perhaps, the design of the Device class. Some things to consider: Is there a good reason why the "poweringOn" state is hidden, while "poweredOn" isn't? What is the side-effect (or, what is the motivation for tracking the "poweringOn" state)? Can you inject a function or an object that will handle any side-effects (that you can then mock with a "spy" (a kind of test double), or test in isolation)? Etc. Jun 18 at 15:30
  • The poweringOn state was there to give the impression that the device was busy doing something while turning on. Its job was to prevent the device from performing some other computation during the simulation. I think the device class we're considering is something that manages animations to its state. So I could create like a PowerOnAnimation which changes state over time and changes the properties of a device. I think this will keep the logic separate and allow me to test the hidden state. Thanks
    – Mrwerdo
    Jun 20 at 2:28
3

Even if the sequence becomes large, it is not necessarily an issue if your test sequence becomes equally large too, as long as

  • the running time isn't bothering you

  • the intermediate states are properly validated, so in case of a failing test, you can identify easily the first step which failed, making it easier to determine the root cause.

In your above case, for example, I would have expected an assertion

  XCTAssertEqual(device.isPoweredOn, false)

between the first and the second device.update().

However, if there are multiple different tests all based on the same introductory "update" sequence, it will make sense to refactor that kind of initializing into a method on its own, so one can reuse it in many different tests. Let's call that method InitDeviceToStateFoo(device), where "StateFoo" has to be replaced by a meaningful word describing the kind of state.

This already will cause your individual tests to be small and maintainable, following the "Arrange, Act, Assert" pattern.

Next step I would recommend is to check wether InitDeviceToStateFoo is as good as it is, or if there are needs to optimize it:

  • optimizing for simplicity or robustness: maybe to reach "state Foo", there is not this complex - potentially brittle - update sequence required, some direct assignments or method calls might be sufficient?

  • optimizing for performance (maybe the complex update sequence is lasting too long, and a direct assignments can make it faster?)

In case there are performance or stability problems with the current API of Device, this sometimes may lead to some design changes to make testing easier. For example, it could make sense to have an option to persist the intermediate state of a device, and reload it at a later point in time, even if that mechanism is exclusively used in testing. Or, there might be a new Device method which sets a Device object directly into a specific state with one call (of course, this method should have a unit test on its own, but then it can be reused in several test).

So what about "when the simulation depends upon time"? Well, this is a pretty different question, and the usual approach is here is mock out any time-related internal components from the Device, like a clock / timeout mechanism, so your unit tests become independent from the temporal situation.

1
  • Thanks. You make some useful points here, particularly about identifying the points that fail. I was concerned about decreasing clarity by adding lots of assertions in between updates but the advantage that you point out is knowing where it failed.
    – Mrwerdo
    Jun 20 at 2:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.