Where does test fixture code fit in?

Question

This is an unusual question since I don't believe it lends itself to typical software development practices but I hope you can prove me wrong.

In MATLAB, my team have created several signal processing components, where data goes in one end, and different data comes out the other. What each of the components do is unimportant at this stage. What matters is that the typical testing pattern my team have adopted is that they write a script which generates the input data (each around 40MB in size), and then they run each test file through the applicable component (scripted). How they determine that their component passes the test or not is complete signal processing wizardry to me but lets assume its correct.

Typically, test data is generated only once and takes about half a day to do so. The test themselves can take anywhere up to day to run.

Coming from more of an application development background, and being given the task of integrating each component into a so-called "reference chain", I need to try and standardize testing and make it fool proof. One thing I wanted them to do is to use the unit testing framework to wrap their tests up into. The unit testing framework in MATLAB is, for the purposes of this question, just like JUnit.

My question is where does the scripts that generate the test data, and the test data itself fit in to the test ecosystem. Obviously the test data is a fixture, and can be treated as such, but I am a bit nervous about where the scripts that generate the test data fit in.

I would also welcome any critiques of my team's testing approach.

Answer 1

First, I think it's good to state what's important. I'm familiar with "continuous integration" types of build systems, so I think that 1) Checking in everything necessary to build, 2) Building from a remote system in an automated way to prove #1, and 3) Testing in automated fashion - are all important. If you have different values, you may come to a very different conclusion.

Based on those values, I see the test data as a dedicated build target, along with one for building the MATLAB code, and one "overall" target for building both and running the test. This doesn't mean that the test data needs to be regenerated every time - something like "make" may be better suited for this type of checking. Since the tests themselves take a day to run, the initial hit of half a day to generate the data may not even be so bad with respect to the big picture. And of course if needed the test data itself can be checked in as a build artifact but you may want to build from scratch every time you do a release.