2

My motivating case is hardware test automation. We have several different testers (benches of lab equipment that connect to the "device under test" (DUT)), and we write test configuration files which instruct the tester what series of tests to perform. The tests performed take in parameters from the configuration as well.

At the moment the coupling between these is higher than I'd like, so I'm trying to find a good pattern to keep the logic for how we perform tests ("tester" object(s)) separate from what tests we wish to perform in any particular run (the configuration objects).

The best I've come up with so far is as follows (syntax not verified, just getting the idea across):

class TestStepConfig
{
    string Name;
    // Parameters are tuple of parameter name to parameter value. If parameter is not truly a string, it will be converted to int, float, etc later.
    IReadOnlyList<Tuple<string, string>> Parameters;
}

interface ITester
{
    // Return value is any data coming back from the test. First tuple element is result name, second is result value (I.E. "Voltage","5.0")
    List<Tuple<string, string>> RunTestStep(TestStepConfig config);
}

class NaiveSequencer
{

    List<TestStepConfig> configs;
    ITester tester;

    void RunFullTest()
    {
        foreach (var config in configs)
            tester.RunTestStep(config);
    }

}

The idea is that each tester has a set of atomic test steps it can run. It knows these steps by name, and the steps take in a set of parameters as per the Parameters field of the TestStepConfig. So, you pass in a TestStepConfig with the name of the desired step and the desired parameters, and the tester performs that test step.

Thus, all any configuration scheme needs to do is determine which TestStepConfigs to feed into the tester, and all the tester needs to understand is how to execute these atomic TestSteps.

What has me wary of this solution is that the "TestStepConfig" looks uncomfortably like re-implementing a method call ("Name" of method, "Parameters" to pass in). I don't want to implement the testers as an actual set of specific "test this, test that" methods because:

  • There may be many, many such test steps available (anywhere from 20 to over 100), leading to extremely large, rigid objects.
  • Many of these tests may or may not be available depending on the particular configuration of the bench (what equipment is available, etc).
  • How a TestStep of a particular name is executed may change
    depending on the setup (what part is being tested, etc)
  • I need to be able to perform sequences of tests with many different testers, and I can't be re-writing the sequencing algorithms every time there's a new tester just because it has a different interface

For these reason and more, the TestSteps I feel are much better modeled as a group of something similar to a Command Pattern (but with execute-time parameters) brought together by a factory and placed into a generic implementation of a ITester object, rather than modeled as compile-time methods of the Tester.

Has anyone had to deal with a case like this before? Is there a better pattern than the one I've proposed above?

2

Has anyone had to deal with a case like this before?

Your bench/test configuration is more complex than I've dealt with, but it is the same problem, just more layers of construction.

Warning: Do not get caught up in inventing framework to genericise and abstract any structure, any test, any time.


Identify, classify, and organize every test

Whether a test is a single method or a series of methods, or a class composite, they're all tests and each test should be explictly identifiable.

Each test should be designed to run by itself. It may run as part of a series of consecutive tests but each identifiable test must be fundamentally independent.


Coding too soon

Thinking in terms of "methods" is too narrow, too sub-atomic as it were. That is why you're seeing "the coupling between these is higher than I'd like." Get the logical organization and the code framework will emerge from that.


Test Structure Observations

  • TestBenches - create a config section.
  • TestBench is probably the top of the structure. This can also act as a namespace, tests in different testbenches can have the same name.
  • Structure will be directly reflected in the config file elements.
  • Test order in the config file is execution order.

Factory Observations

  • You will have an abstract factory. You may have the builder pattern depending on the dynamics of constructing any given test.
  • The abstract factory structure will map directly to the config file structure. It knows the structure. It will necessarily be tightly coupled to the config file.
  • Config element names will be converted to enums. The various enums will reflect the config structure. For example TestBenchXFactory will use a Xtests enum. This effectively validates the XML elements before we attempt to use it.

OO observations

  • Lots of passing the buck but it is all dead simple code until we get to construction of concrete, dynamic test objects.
  • The number of factories may seem excessive or totally unnecessary. However, We're minimizing coupling of layer construction
  • Adding, removing tests may seem like Open/Closed principle violation, but "frankly my dear, I don't give a damn."
  • Good adherence to single responsibility principle in the factory classes makes change easy. Again, Each logical test is independent

<BenchTests>
    <!- BenchTestFactory gets <BenchTests> as a parameter -->
    <!- builds a BenchTestCollection. Instantiate sBenchTestXFactory, etc. -->
    <!- Adds concrete `BenchTest`s to the collection. -->
    <TestBenchX>
        <!- `BenchTestXFactory` gets `<TestBenchX>` element as a parameter --> 
        <!- instantiates `BenchTestX` object. Builds each test, adds to `BenchTestX`. --> 
        <!- Certain tests may need their own builder -->
        <Test1 param1 = 2 param2 = "Hello".. />
        <Test2 />
        <ComplexTest ... >
            <Test1>
            <Test5>
        </ComplexTest>
    </TestBenchX>

    <TestBenchY>
    </TestBenchY>
</BenchTests>

The classes

From the above looks like we have:

public class testBenchCollection {
    List<TestBench> TestBenches;

    public void AddBench ( TestBench newBench ) {}

    public void Execute(){
        foreach( TestBench bench in testBenches )
            bench.Execute();
    }
}

public abstract class TestBench {
    List<Test> testCollection;

    public virtual Execute(){
        foreeach( Test aTest in testCollection )
            aTest.Execute();
    }

    public void AddTest( Test newTest )
}

public abstract class Test {
    public virtual Execute(params object[] myParms ){}
}

public enum TestBench{ X, Y, Z }
public enum GlobalTests { eenie, meenie, minee, moe }
public enum XTests { test1, testToo }
public enum YTests { Larry, Moe, CurleyJoe, test1 }

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.