New sequential testing framework with the same rigor as xUnit?

Question

Looking for books (published or upcoming) and frameworks (preferably free/open-source) for sequential tests.

My understanding of sequential tests is that it is very similar to a Workflow System, with the following elements:

• A collection of Setup/Exercise/Verify routine-sets (suites) - same as in xUnit
  • Optionally, a routine-set may also contain:
    • Teardown - which marks the end of the sequential test, after which the entire software state will be unrolled to the beginning.
    • Undo - which rolls back just a single operation, back to before the "Setup" of the current routine-set.
    • If a programmer provides an Undo, the programmer vouches for the correctness of that Unfo. (Assertions can be used, though it is best not to use any unreliable Undo.) If programmer is not certain, don't provide an Undo.
• Each routine-set must either provide a StartUp, or one or more "preceding step(s)" - the new part for sequential tests
  • A StartUp basically re-initializes the entire software state in a guaranteed way.
  • The "preceding steps" are references to other routine-sets.
• Unlike unit testing, sequential tests are allowed to pass on objects to the next step, in a carefully-controlled way.

Example:

• Routine A
  • Create an empty "readme.txt". (stop the test if it already exist; preconditions not met.)
  • Verify "readme.txt" exists.
  • Undo function: Delete "readme.txt" if it exists.
• Routine B,
  • follows routine A
  • Open file "readme.txt" for writing (erase previous content)
  *

Related: Are scenario tests groups of sequential unit tests?

---

Dear all, sorry the question was lost in the midst of an unexpected machine failure; so this was recovered from a previous autosave draft. Since that crash I had extended the idea a little bit, and I'll need several days to clean it up and update this question.

---

The keywords are:

• Initial [new] (initializes the whole world)
• Setup (checks for pre-conditions are met for the beginning of one step, and create temporary objects that help with this one step - these temporary objects will cease when this one step finishes. You may call these "local" objects)
• Exercise
• Verify (assertions that the step has been successful)
• Undo [new] This optional method, if provided, means that there is a way to fully revert the world state to before this step. If one is provided, the programmer vouches for the correctness of that Undo. If unsure, don't provide one.
• Final [new] Wraps up the whole world for disposal. After that, Initial can be called again to re-initialize a second new world.
• Follows [new] this defines how steps can be chained together.
  • Each test step that is not an Initial step must have one or more Follows statement. Each statement can include one or more peer steps - those peer steps need to happen first before this step can be invoked.

Now the gist. Once you specify all those rules, the sequential test execution engine enumerates ALL of possible sequencing of those sets of steps. It enumerates all paths from Initial to Final, using whatever intermediate steps that allows a path to be formed. It will also warn you about loops and other potential problems, and let you choose how to deal with those loops.

Answer 1

I don't think you can get "all" possible sequencing of a set of steps for non-trivial cases (since you essentially end up with a big n choose k problem that's intractable after a certain number of items).

However, I do recall a framework developed internally at one of my previous employers that employed a genetic algorithm to first exercise a large set of combinations of steps, then identify certain categories of failures (mostly crashes, but theoretically could identify other kinds of failures with the right "driver"), which it would then reduce to an approximate minimum set of steps that reliably reproduced the failure, also using a genetic algorithm. The main complication in this setup was that it's somewhat rare to be able to write tests that contain individual steps whose interaction doesn't create a different expected result, in the real world, so that was part of the reason the framework primarily focused on crashing failures. You could certainly go this route, although some of the work that was done may be subject to certain patent protections in the US. I haven't been connected to that project for about 10 years, but I do know that some patent(s) were applied for, and that some research papers were published about the approach, but I don't know much beyond that.

If you're mostly looking for something that at least allows strict ordering of tests in a unit testing framework, with some kind of dependency resolution, I think TestNg provided such a facility. Data driven tests provide a partial solution to some of your other requirements. Additionally, some test frameworks support a transactional model, which may solve some of your needs; I've written some test cases that relied on an existing transaction mechanism in the language, although this turned out to be problematic when dealing with multiple data access mechanisms in the same application stack that weren't really in the same TransactionScope because the processes executing the transaction were different.