Why are test frameworks like JUnit or TestNG not more "object-oriented"?

Question

By "more object-oriented", I mean, it appears to me testing frameworks like TestNG and JUnit could encourage testers to write implementations of Test and TestSuite interfaces. The current approach using @Testannotations often leads to huge classes with lots of methods and lots of boilerplate code.

A more "OO" way could, for example, look like this:

public class MyTestSuite extends framework.AbstractTestSuite {

  @Override
  public void setup() {
    //...
  }

  @Override
  public void execution() {
    executeInParallel(
      new MyTestA("some param"), 
      new MyTestA("some other param"),
      new MyTestB());
    execute(new MyTestBSubclass());
  }

  @Override
  public void teardown() {
    //...
  }
}

Are there problems with this approach I'm not anticipating?

Edit: My assumption here is that the framework.Test interface would have a single test method, i.e.

interface Test {
  void exec();
}

with the understanding that a Test fails if and only if an exception is thrown (alternatively, the method could return a boolean).

The thing that feels a bit funny to me about JUnit/TestNG is that the methods seem to be treated as nouns rather than verbs. If we use instances as tests, I would think we'd be able to take advantage of things like constructor parameterization, subclassing, and composition, and these would more than make up for creating a class for each sort of test. And with lambdas, we could probably get by without creating those, a lot of the time.

But there may be disadvantages I'm not thinking of...

Answer 1

The number one fundamental challenge with test frameworks is that it is not an object oriented problem. Initially, JUnit 1 was much more object oriented:

• There was a TestSuite object where you declared all the test objects you wanted to run at once, and a callback to start the test
• The Test base class had a setUp() and tearDown() virtual method you could override to perform that function.

The real problem was the individual tests. You could have a virtual method where you declare all the methods that need to be called, but then the usefulness of the framework starts to be lost. On day one, JUnit had to use reflection to be useful.

JUnit had the basic pattern that if you prepended your test method with the word test it would get executed as a test. A typical test class would be something like this:

public class MyBeautifulTest extends Test {
    public void testThisIsATest() {
        Assert.isTrue(true);
    }
}

This worked well enough for a long time, but a number of people complained about the names of the methods used to set up and tear down. They complained that requiring the base class Test prevented different types of test organization due to the base class.

Enter the Annotation

Eventually, the Java language came out with the Annotation feature. This was a means to avoid all the complaints against JUnit. It was at this time that TestNG was born. TestNG was built with the intention of making use of the new language features, and allowing users the ability to name the setup/teardown methods whatever they wanted. No base class meant that users could assemble their tests however they wanted.

Eventually JUnit adapted because that is where the demand was. Both JUnit and TestNG scanned for test classes with the appropriate annotations so that users no longer had to hand maintain the test suite. That meant you only needed to create a new test class with at least one test and it would automatically get picked up.

Long Story Short

The use of annotations and static method imports improved the usability of the test frameworks. It also opened up the opportunity for new ways of testing, for example:

• Parameterized tests (passing in values to perform the same test on a range of values)
• Theories (testing "truth" across your application)
• Categorizing tests for better reporting

You might be able to imagine how all this would work in a pure OO manner, but with the design of the Java language you will always have some compromises. You most definitely won't be able to match the utility of these test frameworks without the use of reflection. Other object oriented languages may have different facilities that enable the utility in a pure OO design. However Java (and C#) have limitations where Annotations, reflection, and fluent assertions provide the most utility for the least amount of code that the developer has to maintain.

At the end of the day, if it is painful to maintain your tests, they won't get written. It's already an uphill battle to get many developers to automate unit tests. Every barrier demotivates them further.

Answer 2

You asked

Are there problems with this approach I'm not anticipating?

An interface in Java or C# provides you only with a finite, fixed number of methods a class can implement. So your suggestion makes it quite hard (if not impossible) to separate the individual tests by reflection. For example, the JUnit GUI would not be able to display the list of available tests before they are executed, and could not easily offer options for running them individually.

Answer 3

The JUnit test framework started a direct port of Kent Beck's SUnit framework/pattern [1][2] that arose roughly a decade earlier in the Smalltalk/Agile community. SUnit does feature individual test objects. However, not all of that maps cleanly to the Java that was available in the 90s. In particular, whereas SUnit uses selectors, JUnit uses reflection and later annotations to discover test cases.

A fairly literal port of SUnit would look like this in Java. First, we define a TestCase class. Instances of this class are test cases that can be run(), similar to what you suggest. Methods of subclasses can be used to create those test cases, which are resolved by reflection. Each TestCase subclass represents a fixture – an environment in which test cases run. This fixture is configured by the setUp() method.

class TestCase {
  private String selector;

  public TestCase(String selector) {
    this.selector = selector;
  }

  /// Run whatever code you need to get ready for the test to run.
  protected void setUp() { }

  /// Release whatever resources you used for the test.
  protected void tearDown() { }

  /// Run the selected method.
  public final void run() throws Throwable {
    setUp();
    try {
      this.getClass().getMethod(selector).invoke(this);
    } catch (java.lang.reflect.InvocationTargetException e) {
        throw e.getCause();  // unwrap wrapped exception
    } finally {
      tearDown();
    }
  }

  /// Run the selected method and register the result.
  public final void run(TestResult result) {
    try {
      run();
    } catch(Throwable e) {
      result.error(this, e);
      return;
    }
    result.pass(this);
  }

  /// The base class can define assertions that will be inherited by tests.
  protected void should(boolean result) {
    if (!result) throw new AssertionError("check failed");
  }

  protected void shouldnt(boolean result) {
    if (result) throw new AssertionError("check failed");
  }

  @Override public String toString() {
    return String.format("%s.%s", getClass().getName(), selector);
  }
}

A TestSuite is a collection of test cases:

class TestSuite {
  public final String name;
  private List<TestCase> testCases = new ArrayList<>();

  public TestSuite(String name) {
    this.name = name;
  }

  public TestSuite addTestCase(TestCase testCase) {
    testCases.add(testCase);
    return this;
  }

  public TestResult run() {
    TestResult result = new TestResult(name);
    for (TestCase each: testCases) {
      each.run(result);
    }
    return result;
  }
}

class TestResult {
  public final String name;
  public List<String> errors = new ArrayList<>();
  public List<String> passed = new ArrayList<>();

  public TestResult(String name) {
    this.name = name;
  }

  public void error(TestCase testCase, Throwable error) {
    errors.add(String.format("%s: %s", testCase, error));
  }

  public void pass(TestCase testCase) {
    passed.add(testCase.toString());
  }

  public int count() {
    return passed.size() + errors.size();
  }
}

With that, we can implement an example test suite:

class SetTestCase extends TestCase {
  Set<Object> empty;
  Set<Object> full;

  public SetTestCase(String selector) {
    super(selector);
  }

  @Override protected void setUp() {
    empty = new HashSet<>();
    full = new HashSet<>();
    full.add("abc");
    full.add(5);
  }

  public void testAdd() {
    empty.add(5);
    should(empty.contains(5));
  }

  public void testRemove() {
    full.remove(5);
    should(full.contains("abc"));
    shouldnt(full.contains(5));
  }

  public static TestSuite testSuite() {
    TestSuite suite = new TestSuite("Set Tests");
    suite.addTestCase(new SetTestCase("testAdd"));
    suite.addTestCase(new SetTestCase("testRemove"));
    return suite;
  }

  public static void main(String[] args) {
    TestResult result = testSuite().run();
    System.out.printf("Ran %d test cases\n", result.count());
    for (String error: result.errors) {
      System.out.printf("Test failure: %s\n", error);
    }
    if (!result.errors.isEmpty())
      System.exit(1);
  }
}

This approach is quite flexible: you can run test methods directly (Smalltalk was an inherently interactive environment). You can add arbitrary logic when assembling a test suite. But it has some problematic aspects:

• test cases are resolved by name
• test cases must be added manually to a suite – that's very error-prone and tedious
• test case fixtures aren't reused
• construction/set up of test cases happens in separate phases
• creating one-off fixtures is tedious in Java

JUnit fixes some of these problems, importantly that test cases are discovered via reflection and don't have to be listed explicitly. On the other hand this sacrifices some flexibility. Because a TestCase class contains the code of multiple cases, it continues to be misused as a test case collection rather than a fixture with some test cases running in that fixture. But perhaps the setUp() method isn't the best approach for managing fixtures?

Note that TestCase cannot be an interface because in this design it supplies some plumbing for method resolution, result management, and provides assertions to the subclasses. However, there could be an additional interface that hides everything but run(TestResult) from the TestSuite.

In languages that, unlike Java or Smalltalk, do not enforce a strict class-oriented structure of the code, it is indeed possible to get drastically simpler test structures that focus on the behavior of the classes rather than on simple set-up of fixtures. E.g. JavaScript favours Rspec-style describe-it tests:

describe('Set', () => {

  it('can add elements', () => {
    const empty = new Set();
    empty.add(5);
    assert(empty.has(5));
  });

  it('can delete elements', () => {
    const full = new Set(["abc", 5]);
    full.delete(5);
    assert(full.has("abc"));
    assert(!full.has(5));
  });

});

You can't pull that off in Java without having one gargantuan method for all your test cases. And you literally couldn't do that before Java 8 gave us lambdas. You also lose some benefits, for example being able to implicitly set up fixtures (though most implementations of this pattern provide a before method).

The point is, a lot has happened since the xUnit pattern was popularized:

• We've learned a lot more about OOP. The best practices of yesteryear aren't always the best practices of today. OOP is not always a good fit. Java's class-oriented approach to programming is restrictive and prevents better alternatives. Where JUnit sticks to a certain design, it is also motivated by backwards compatibility.

• We've learned a lot more about automated testing. The xUnit style to unit testing is no longer state of the art – it is neither good for capturing requirements nor convenient for the programmer. Better approaches include BDD-style techniques such as specification by example or Rspec-style tests, and property-based testing à la Haskell's quickcheck. Some of that is available in Java and even JUnit, but it must still somehow fit into Java's structure – which often means methods with magic names or annotations.

Note that Java and C# aren't the only languages that make elegant testing more difficult. E.g. Python's pytest has a much better fixture system than xUnit can ever dream of, but still suffers from the inflexibility of the host language (no real lambdas). One of the most useful approaches that I've found is to treat testing not as a framework but as a pattern: where the standard test frameworks get in your way, it is often more convenient to create your own quick test implementation that perfectly fits your style of testing.