Building automated unit tests for tools which don't have an xUnit implementation

Question

Preamble

I have been bitten by the bug of automated unit testing. I feel the benefits and the confidence in a code base it can deliver.

I am also feel I have a reasonable intuition as to what parts of the code deserve to be unit tested. Code that has logic, code that might be messy (because it's dealing with some messy requirements) that you are happy to have encapsulated, that may have a few odd fringe conditions.

Unusual Context

At my place of work, we leverage a number of tools which are fairly niche. For the sake of argument I can certainly see that these tools deliver on features and efficiencies making them indispensable for getting their particular job done.

The challenge I have is that I certainly have implemented logic in these tools which my gut says "this should be covered with a unit test".

I have tested these bits of code...

• In a throwaway fashion - try several inputs to exercise the code until satisfied it works - this has the downside of being lost effort when it comes to people maintaining the code in the future
• In a less throwaway but manual fashion - writing up the test cases and steps to poke the code to produce output; with test data; such that the above throwaway tests are not thrown away - this retains the investment in time, but is slow and subject to people knowing about the tests and running them
• In a slightly more automated fashion - with scripts to do the poking of the code; perhaps even check the output to a degree - this is more pleasant to re-run but is far from leveraging a mature framework in a consistent manner like when writing the xUnit style tests

While the progression feels nice, I feel I'm missing the xUnit experience I've become a fan of.

What kind of languages/platforms exactly am I dealing with here?

• An ETL tool, akin to SSIS, called RedPoint Data Management: not open source so I'll describe, it has projects which have tools, these tools are linked by data flows, and some have some logic I wish to unit test

• A composition tool, this for the uninitiated is a product which takes a data file and outputs a print file. Imagine XML in, PDF out.

• DOS script functions: I promise this is a real feature of good'ol'fashioned DOS scripts, tutorial here

• (I'll add this to the list, not strictly what I need help with, but may be more clear than the above if my tool set is difficult to empathise with) COBOL data transformations, using CopyBooks to specify the formats of the flat file inputs and outputs. When compiled results in one big-ball-of-mud binary even if functionally decomposed inside

The Actual Question

Are xUnit style tests fundamentally aimed at Object Oriented languages?

Answer 1

The distinction is not about object-oriented or not, but rather about testing individual functions ("white box testing") versus complete programs ("black box testing"). And it's not so much separate cases as a continuum in which various techniques are mixed and matched.

The tools are also not monolithic. An testing framework consists of four things:

• The test runner
• The result reporter
• The collection of assertions
• (optionally) Mock object framework

These components are mostly orthogonal and independent implementations of them can be combined, both the different types of components and multiple implementations of each one.

The test runner

The component that finds all test cases and executes them. Multiple test runners can be combined together by simply calling a test runner as test case from another runner. Can be as simple as a for-loop over a directory of scripts and that's exactly what one often ends up using as top-level runner when combining tests written in different frameworks.

You can find some universal test runners in CTest or shUnit. Both also provide methods for comparing outputs in files.

The perl Test::Harness can also be considered general runner as it runs scripts in directory. And perl is well suited for testing external programs too.

The reporter

Each unit test framework produces list of failed tests. If you combine multiple frameworks, you may either just collect all individual reports or use something like subunit or the Test::Harness's "tap" to provide nice uniform progress and summary.

The assertions

This is how you test the tested code produces expected results and is tied to how the code is interfaced.

• For libraries in various programming languages there are the assertions provided by the respective xUnit.
• For command-line tools there are various file comparators. CTest and shUnit have some, but they are mostly easy to homebrew if needed.
• For interactive command-line tools there is expect (or it's Perl variant).
• For GUI applications there is LDTP (linux desktop testing project, but they now have windows and macos versions too)
• I am sure I've seen something for web applications (mean in-browser ones; you test services through their client library), but I don't remember any name. You'll surely find a few with google.

Mocking tool

This is about constructing the test environment without setting up all real dependencies. E.g. set up a database access layer that will return specified set of responses without setting up actual database and such.

It's not possible to automate much as you need to provide the data it should return. It is often not what you want anyway. I.e. you need to test the application against sample data in the actual database or other data source anyway and additional fine-grained module testing is often a lot of work with diminishing returns.

I don't think much can be generalized for this. For most xUnits there is something that will generate dummy implementations for interfaces, but that's about it. Than you simply have to consider what the component to be tested requires and what's the easiest way to provide it locally. For something that communicates with some web service you may run some lightweight http server locally. For database tool that uses ODBC (or DBI or whatever universal API) you can test it against sqlite even though it normally uses big server etc. For tools that just read files you obviously just copy in suitable test data set.

Answer 2

I'm not entirely sure what sort of tools you are talking about here or what makes them difficult to test.

But might it be worth taking a look at ApprovalTests? This is a cross-language (Java, C#, VB.Net, PHP, Ruby) framework which sits underneath any unit test runner. It takes the approach of: instead of writing and test and specifying the expected result, you write the test, don't specify a result, run the test, and then it shows you the result. You approve it. From that point on, every time you run the test, it will pass if the result is unchanged from when you approved it, and fail (presenting you with a diff) if it has changed.

This is great for testing code that returns more than just simple scalar results (collections, XML documents, etc.) And because it runs under any of the major unit test frameworks, you don't lose any of your tooling around automation, etc.