This is an unusual question since I don't believe it lends itself to typical software development practices but I hope you can prove me wrong.

In MATLAB, my team have created several signal processing components, where data goes in one end, and different data comes out the other. What each of the components do is unimportant at this stage. What matters is that the typical testing pattern my team have adopted is that they write a script which generates the input data (each around 40MB in size), and then they run each test file through the applicable component (scripted). How they determine that their component passes the test or not is complete signal processing wizardry to me but lets assume its correct.

Typically, test data is generated only once and takes about half a day to do so. The test themselves can take anywhere up to day to run.

Coming from more of an application development background, and being given the task of integrating each component into a so-called "reference chain", I need to try and standardize testing and make it fool proof. One thing I wanted them to do is to use the unit testing framework to wrap their tests up into. The unit testing framework in MATLAB is, for the purposes of this question, just like JUnit.

My question is where does the scripts that generate the test data, and the test data itself fit in to the test ecosystem. Obviously the test data is a fixture, and can be treated as such, but I am a bit nervous about where the scripts that generate the test data fit in.

I would also welcome any critiques of my team's testing approach.

  • I don't think this is actually that odd of a use case. Two questions to help anyone answering: 1) Does test data get reused between tests? 2) How long does it take to regenerate the test data? – J Trana Jan 6 '15 at 3:30
  • I've updated the answer @JTrana, thanks for the comment – Sam Delaney Jan 6 '15 at 7:12

First, I think it's good to state what's important. I'm familiar with "continuous integration" types of build systems, so I think that 1) Checking in everything necessary to build, 2) Building from a remote system in an automated way to prove #1, and 3) Testing in automated fashion - are all important. If you have different values, you may come to a very different conclusion.

Based on those values, I see the test data as a dedicated build target, along with one for building the MATLAB code, and one "overall" target for building both and running the test. This doesn't mean that the test data needs to be regenerated every time - something like "make" may be better suited for this type of checking. Since the tests themselves take a day to run, the initial hit of half a day to generate the data may not even be so bad with respect to the big picture. And of course if needed the test data itself can be checked in as a build artifact but you may want to build from scratch every time you do a release.

  • Thanks for the answer, I'll give it ago and get back to you. The problem at the moment is that they've decided to keep the code that generates the test data together with the code which runs the test. One of the big draw backs is that we can't have a continuous integration system so we have to rely on human beings who are notoriously unreliable (myself included). – Sam Delaney Jan 9 '15 at 21:32
  • Yeah, I understand. At the very least make really good documentation around how to regenerate the test data or your future self may be quite confused (found myself in this sort of situation one too many times...) – J Trana Jan 10 '15 at 1:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.