Looking for books (published or upcoming) and frameworks (preferably free/open-source) for sequential tests.
My understanding of sequential tests is that it is very similar to a Workflow System, with the following elements:
- A collection of Setup/Exercise/Verify routine-sets (suites) - same as in xUnit
- Optionally, a routine-set may also contain:
- Teardown - which marks the end of the sequential test, after which the entire software state will be unrolled to the beginning.
- Undo - which rolls back just a single operation, back to before the "Setup" of the current routine-set.
- If a programmer provides an Undo, the programmer vouches for the correctness of that Unfo. (Assertions can be used, though it is best not to use any unreliable Undo.) If programmer is not certain, don't provide an Undo.
- Optionally, a routine-set may also contain:
- Each routine-set must either provide a StartUp, or one or more "preceding step(s)" - the new part for sequential tests
- A StartUp basically re-initializes the entire software state in a guaranteed way.
- The "preceding steps" are references to other routine-sets.
- Unlike unit testing, sequential tests are allowed to pass on objects to the next step, in a carefully-controlled way.
Example:
- Routine A
- Create an empty "readme.txt". (stop the test if it already exist; preconditions not met.)
- Verify "readme.txt" exists.
- Undo function: Delete "readme.txt" if it exists.
- Routine B,
follows
routine A- Open file "readme.txt" for writing (erase previous content) *
Related: Are scenario tests groups of sequential unit tests?
Dear all, sorry the question was lost in the midst of an unexpected machine failure; so this was recovered from a previous autosave draft. Since that crash I had extended the idea a little bit, and I'll need several days to clean it up and update this question.
The keywords are:
Initial
[new] (initializes the whole world)Setup
(checks for pre-conditions are met for the beginning of one step, and create temporary objects that help with this one step - these temporary objects will cease when this one step finishes. You may call these "local" objects)Exercise
Verify
(assertions that the step has been successful)Undo
[new] This optional method, if provided, means that there is a way to fully revert the world state to before this step. If one is provided, the programmer vouches for the correctness of that Undo. If unsure, don't provide one.Final
[new] Wraps up the whole world for disposal. After that, Initial can be called again to re-initialize a second new world.Follows
[new] this defines how steps can be chained together.- Each test step that is not an
Initial
step must have one or moreFollows
statement. Each statement can include one or more peer steps - those peer steps need to happen first before this step can be invoked.
- Each test step that is not an
Now the gist. Once you specify all those rules, the sequential test execution engine enumerates ALL of possible sequencing of those sets of steps. It enumerates all paths from Initial
to Final
, using whatever intermediate steps that allows a path to be formed. It will also warn you about loops and other potential problems, and let you choose how to deal with those loops.