Simulation Testing: how to test something that depends on its previous state

Question

I'm practicing test driven development while implementing a network simulator: devices communicate with each other and implement little bits of functionality that respond to messages. Each device has its own view of the network topology and can be individually powered on or off.

Timing is important. If the Device is in the simulation, has powered, and is powered off, it should be powered on after exactly two update calls.

Below is an extract of a test case I have written, trying to decompose the problem. Be warned that to a large degree I'm making the design up as I fly, so it could be the case that I haven't decomposed the objects involved enough.

let device = Device()
device.hasPower = true
device.isPoweredOn = false

// There is hidden state that we do not know about. 
// Does this make it a bad test?
// Is this instead some kind of integration test?
device.update()
XCTAssertEqual(device.log.last, .poweringOn)
device.update()
XCTAssertEqual(device.log.last, .poweredOn)
XCTAssertEqual(device.isPoweredOn, true)

Notice how device depends upon its previous state between the two assert statements.

Taking a step back, what is actually going on is I'm verifying that the state of device is equal to a sequence of state changes:

• .poweringOn, then
• .poweredOn.

The question is: how can I write effective tests when the number of elements in the sequence becomes large? (or, equivalently, when the simulation depends upon time, or equivalently, when you have an update loop).

Research

Why not look at what other people do for testing simulation software. OMNeT++ I'm looking at you.. I'm looking for an approach that doesn't involve setting up external tests. That is, I'd like to tests specific bits of functionality in my codebase because it is convenient with today's tooling and I believe it is more rigourous.

I looked at other questions on this site but couldn't find anything related to testing simulations. I searched for "depends on previous state" and "simulation testing" but couldn't find anything that matched what I was looking for. I also tried googling "multiple assets unit testing", and "simulation testing", but again the results are not relevant to my problem.

Answer 1

Even if the sequence becomes large, it is not necessarily an issue if your test sequence becomes equally large too, as long as

• the running time isn't bothering you

• the intermediate states are properly validated, so in case of a failing test, you can identify easily the first step which failed, making it easier to determine the root cause.

In your above case, for example, I would have expected an assertion

  XCTAssertEqual(device.isPoweredOn, false)

between the first and the second device.update().

However, if there are multiple different tests all based on the same introductory "update" sequence, it will make sense to refactor that kind of initializing into a method on its own, so one can reuse it in many different tests. Let's call that method InitDeviceToStateFoo(device), where "StateFoo" has to be replaced by a meaningful word describing the kind of state.

This already will cause your individual tests to be small and maintainable, following the "Arrange, Act, Assert" pattern.

Next step I would recommend is to check whether InitDeviceToStateFoo is as good as it is, or if there are needs to optimize it:

• optimizing for simplicity or robustness: maybe to reach "state Foo", there is not this complex - potentially brittle - update sequence required, some direct assignments or method calls might be sufficient?

• optimizing for performance (maybe the complex update sequence is lasting too long, and a direct assignments can make it faster?)

In case there are performance or stability problems with the current API of Device, this sometimes may lead to some design changes to make testing easier. For example, it could make sense to have an option to persist the intermediate state of a device, and reload it at a later point in time, even if that mechanism is exclusively used in testing. Or, there might be a new Device method which sets a Device object directly into a specific state with one call (of course, this method should have a unit test on its own, but then it can be reused in several test).

So what about "when the simulation depends upon time"? Well, this is a pretty different question, and the usual approach is here is mock out any time-related internal components from the Device, like a clock / timeout mechanism, so your unit tests become independent from the temporal situation.