continuous integration for scientific software

Question

I'm no software engineer. I'm a phd student in the field of geoscience.

Almost two years ago I started programming a scientific software. I never used continuous integration (CI), mainly because at first I didn't know it exists and I was the only person working on this software.

Now since the base of the software is running other people start to get interested in it and want to contribute to the software. The plan is that other persons at other universities are implementing additions to the core software. (I'm scared they could introduce bugs). Additionally, the software got quite complex and became harder and harder to test and I also plan to continue working on it.

Because of this two reasons, I'm now more and more thinking about using CI. Since I never had a software engineer education and nobody around me has ever heard about CI (we are scientists, no programmers) I find it hard to get started for my project.

I have a couple of questions where I would like to get some advice:

First of all a short explanation of how the software works:

• The software is controlled by one .xml file containing all required settings. You start the software by simply passing the path to the .xml file as an input argument and it runs and creates a couple of files with the results. One single run can take ~ 30 seconds.

• It is a scientific software. Almost all of the functions have multiple input parameters, whose types are mostly classes which are quite complex. I have multiple .txt files with big catalogs which are used to create instances of these classes.

Now let's come to my questions:

1. unit tests, integration tests, end-to-end tests?: My software is now around 30.000 lines of code with hundreds of functions and ~80 classes. It feels kind of strange to me to start writing unit tests for hundreds of functions which are already implemented. So I thought about simply creating some test cases. Prepare 10-20 different .xml files and let the software run. I guess this is what is called end-to-end tests? I often read that you should not do this, but maybe it is ok as a start if you already have a working software? Or is it simply a dumb idea to try to add CI to an already working software.

2. How do you write unit tests if the function parameters are difficult to create? assume I have a function double fun(vector<Class_A> a, vector<Class_B>) and usually, I would need to first read in multiple text files to create objects of type Class_Aand Class_B. I thought about creating some dummy functions like Class_A create_dummy_object() without reading in the text files. I also thought about implementing some kind of serialization. (I do not plan to test the creation of the class objects since they only depend on multiple text files)

3. How to write tests if results are highly variable? My software makes use of big monte-carlo simulations and works iteratively. Usually, you have ~1000 iterations and at every iteration, you are creating ~500-20.000 instances of objects based on monte-carlo simulations. If only one result of one iteration is a bit different the whole upcoming iterations are completely different. How do you deal with this situation? I guess this a big point against end-to-end tests, since the end result is highly variable?

Any other advice with CI is highly appreciated.

Answer 1

Testing scientific software is difficult, both because of the complex subject matter and because of typical scientific development processes (aka. hack it until it works, which doesn't usually result in a testable design). This is a bit ironic considering that science should be reproducible. What changes compared to “normal” software is not whether tests are useful (yes!), but which kinds of test are appropriate.

Handling randomness: all runs of your software MUST be reproducible. If you use Monte Carlo techniques, you must make it possible to provide a specific seed for the random number generator.

• It is easy to forget this e.g. when using C's rand() function which depends on global state.
• Ideally, a random number generator is passed as an explicit object through your functions. C++11's random standard library header makes this a lot easier.
• Instead of sharing random state across modules of the software, I've found it useful to create a second RNG which is seeded by a random number from the first RNG. Then, if the number of requests to the RNG by the other module changes, the sequence generated by the first RNG stays the same.

Integration tests are perfectly fine. They are good at verifying that different parts of your software play together correctly, and for running concrete scenarios.

• As a minimum quality level “it doesn't crash” can already be a good test result.
• For stronger results, you will also have to check the results against some baseline. However, these checks will have to be somewhat tolerant, e.g. account for rounding errors. It can also be helpful to compare summary statistics instead of full data rows.
• If checking against a baseline would be too fragile, check that the outputs are valid and satisfy some general properties. These can be general (“selected locations must be at least 2km apart”) or scenario-specific, e.g. “a selected location must be within this area”.

When running integration tests, it is a good idea to write a test runner as a separate program or script. This test runner performs necessary setup, runs the executable to be tested, checks any results, and cleans up afterwards.

Unit test style checks can be quite difficult to insert into scientific software because the software has not been designed for that. In particular, unit tests get difficult when the system under test has many external dependencies/interactions. If the software is not purely object-oriented, it is not generally possible to mock/stub those dependencies. I've found it best to largely avoid unit tests for such software, except for pure math functions and utility functions.

Even a few tests are better than no tests. Combined with the check “it has to compile” that's already a good start into continuous integration. You can always come back and add more tests later. You can then prioritize areas of the code that are more likely to break, e.g. because they get more development activity. To see which parts of your code are not covered by unit tests, you can use code coverage tools.

Manual testing: Especially for complex problem domains, you will not be able to test everything automatically. E.g. I'm currently working on a stochastic search problem. If I test that my software always produces the same result, I can't improve it without breaking the tests. Instead, I've made it easier to do manual tests: I run the software with a fixed seed and get a visualization of the result (depending on your preferences, R, Python/Pyplot, and Matlab all make it easy to get high-quality visualizations of your data sets). I can use this visualization to verify that things did not go terribly wrong. Similarly, tracing the progress of your software via logging output can be a viable manual testing technique, at least if I can select the type of events to be logged.

Answer 2

It feels kind of strange to me to start writing unit tests for hundreds of functions which are already implemented.

You'll want to (typically) write the tests as you change said functions. You don't need to sit back and write hundreds of unit tests for the existing functions, that would be (largely) a waste of time. The software is (probably) working okay as is. The point of these tests is to ensure future changes aren't breaking the old behavior. If you never change a particular function again, it'll probably never be worth taking the time to test it (since it's currently working, has always worked, and will likely continue to work). I recommend reading Working Effectively With Legacy Code by Michael Feathers on this front. He's got some great general strategies for testing things that already exist, including dependency breaking techniques, characterization tests (copy/paste function output into test suite to ensure you maintain regression behavior), and much much more.

How do you write unit tests if the function parameters are difficult to create?

Ideally, you don't. Instead, you make the parameters easier to create (and therefore make your design easier to test). Admittedly, design changes take time, and these refactorings can be difficult on legacy projects like yours. TDD (Test Driven Development) can help with this. If the parameters are super hard to create, you'll have a lot of trouble writing tests in a test-first style.

In the short term, use mocks, but beware of mocking hell and issues that come with them in the long term. As I've grown as a software engineer, though, I've realized mocks are almost always a mini-smell that are trying to wrap up some bigger problem and not addressing the core issue. I like to refer to it as "turd wrapping", because if you put a piece of tin foil on a bit of dog poo on your carpet, it still stinks. What you have to do is actually get up, scoop the poop, and throw it in the garbage, and then take out the garbage. This is obviously more work, and you risk getting some fecal matter on your hands, but better for you and your health in the long run. If you keep just wrapping those poopies you'll not want to live in your house much longer. Mocks are similar in nature.

For instance, if you have your Class_A that's hard to instantiate because you have to read in 700 files, then you could just mock it. Next thing you know, your mock gets out of date, and the real Class_A does something wildly different than the mock, and your tests are still passing even though they should be failing. A better solution is to break down Class_A into easier to use/test components, and test those components instead. Maybe write one integration test that actually hits disk and make sure Class_A works as a whole. Or maybe just have a constructor for Class_A that you can instantiate with a simple string (representing your data) instead of having to read from disk.

How to write tests if results are highly variable?

A couple of tips:

1) Use inverses (or more generally, property-based testing). What's the fft of [1,2,3,4,5]? No idea. What's ifft(fft([1,2,3,4,5]))? Should be [1,2,3,4,5] (or close to it, floating point errors might come up).

2) Use "known" asserts. If you write a determinant function, it might be hard to say what the determinant is of a 100x100 matrix. But you do know that the determinant of the identity matrix is 1, even if it's 100x100. You also know that the function should return 0 on a non-invertible matrix (like a 100x100 full of all 0s).

3) Use rough asserts instead of exact asserts. I wrote some code a while ago that was registering two images by generating tie points that create a mapping between the images and doing a warp between them to make them match. It could register at a sub-pixel level. How can you test it? Things like:

EXPECT_TRUE(reg(img1, img2).size() < min(img1.size(), img2.size()))

since you can only register on overlapping parts, the registered image must be smaller or equal to your smallest image), and also:

scale = 255
EXPECT_PIXEL_EQ_WITH_TOLERANCE(reg(img, img), img, .05*scale)

since an image registered to itself should be CLOSE to itself, but you might experience a bit more than floating point errors due to the algorithm at hand, so just check each pixel is with +/- 5% of the valid range (0-255 is a common range, greyscale). Should at least be the same size. You can even just smoke test (i.e. call it and make sure it doesn't crash). In general, this technique is better for larger tests where the end result can't be (easily) calculated a priori to running the test.

4) Use OR STORE a random number seed for your RNG.

Runs do need to be reproducible. It is false, however, that the only way to get a reproducible run is to provide a specific seed to a random number generator. Sometimes randomness testing is valuable. I've seen a bugs in scientific code that crop up in degenerate cases that were randomly generated. Instead of always calling your function with the same seed, generate a random seed, and then use that seed, and log the seed's value. That way every run has a different random seed, but if you get a crash, you can re-run the result by using the seed you've logged to debug. I've actually used this in practice and it squashed a bug, so I figured I'd mention it. Downside: You have to log your test runs. Upside: Correctness and bug nuking.

HTH.

Answer 3

1. Types of test

   • It feels kind of strange to me to start writing unit tests for hundreds of functions which are already implemented

     Think of it the other way round: if a patch touching several functions breaks one of your end-to-end tests, how are you going to figure out which one is the problem?

     It's much easier to write unit tests for individual functions than for the whole program. It's much easier to be sure you have good coverage of an individual function. It's much easier to refactor a function when you're sure the unit tests will catch any corner cases you broke.

     Writing unit tests for already-existing functions is perfectly normal for anyone who has worked on a legacy codebase. They're a good way to confirm your understanding of the functions in the first place and, once written, they're a good way to find unexpected changes of behaviour.

   • End-to-end tests are also worthwhile. If they're easier to write, by all means do those first and add unit tests ad-hoc to cover the functions you're most concerned about others breaking. You don't have to do it all at once.

   • Yes, adding CI to existing software is sensible, and normal.

2. How to write unit tests

   If your objects are really expensive and/or complex, write mocks. You can just link the tests using mocks separately from the tests using real objects, instead of using polymorphism.

   You should anyway have some easy way of creating instances - a function to create dummy instances is common - but having tests for the real creation process is also sensible.

3. Variable results

   You must have some invariants for the result. Test those, rather than a single numerical value.

   You could provide a mock pseudorandom number generator if your monte carlo code accepts it as a parameter, which would make the results predictable at least for a well-known algorithm, but it's brittle unless it literally returns the same number every time.

Answer 4

1. It is never a dumb idea to add CI. From experience I know this is the way to go when you have an open source project where people are free to contribute. CI allows you to stop people from adding or changing code if the code breaks your program, so it is almost invaluable in having a working codebase.

   When considering tests, you can certainly provide some end-to-end tests (i think it is a subcategory of integration tests) to be sure that your code flow is working the way it should. You should provide at least some basic unit tests to make sure that functions output the right values, as part of the integration tests can compensate for other errors made during the test.

2. Test object creation is something quite difficult and laboursome indeed. You are right in wanting to make dummy objects. These objects should have some default, but edge case, values for which you certainly know what the output should be.

3. The problem with books about this subject is that the landscape of CI (and other parts of devops) evolves so quickly anything in a book is probably going to be out of date a few months later. I do not know of any books that could help you, but Google should, as always, be your saviour.

4. You should run your tests yourself multiple times and do statistical analysis. That way you can implement some test cases where you take the median/average of multiple runs and compare it to your analysis, as to know what values are correct.

Some tips:

• Use the integration of CI tools in your GIT platform to stop broken code from entering your codebase.
• stop merging of code before peer-review was done by other developers. This makes errors more easily known and again stops broken code from entering your codebase.

Answer 5

In a reply before amon already mentioned some very important points. Let me add some more:

1. Differences between development of scientific software and commercial software

For scientific software, the focus normally is on the scientific problem, of course. The problems are more handling the theoretical background, finding the best numerical method, etc. The software is only one, more or less, small part of the work.

The software is in most cases written by one or only a few persons. It is often written for a specific project. When the project is finished and everything is published, in many cases the software is no longer needed.

Commercial software is typically developed by large teams over a longer period of time. This requires a lot of planning for architecture, design, unit tests, integration tests etc. This planning requires a substancial amount of time and experience. In a scientific environment, there normally is no time for that.

If you want to convert your project to a software similar to commercial software, you should check the following:

• Dou you have the time and resources?
• What is the long-term perspective of the software? What will happen with the software when you finish your work and are leaving the university?

2. End to end tests

If the software gets more and more complex and several persons are working on it, tests are mandatory. But as amon already mentioned, adding unit tests to scientific software is quite difficult. So you have to use a different approach.

As your software is getting its input from a file, like most scientific software, it is perfect for creating several sample input and output files. You should run those tests automatically on each release and compare the results against your samples. This could be a very good replacement for unit tests. You get integration tests this way as well.

Of course, to get reproducible results, you should use the same seed for your random number generator, as amon already wrote.

The examples should cover typical results of your software. This should also include edge cases of your parameter space and numerical algorithms.

You should try to find examples which do not need too much time for running, but still cover typical test cases.

3. Continous integration

As running the test examples may take some time, I think that continous integration is not feasible. You probably will have to discuss the additional parts with your colleagues. For example, they have to match to the numerical methods used.

So I think it is better doing the integration in a well-defined way after discussing the theoretical background and numerical methods, carefull testing, etc.

I do not think that it is a good idea having some kind of automatism for continous integration.

By the way, are you using a version control system?

4. Testing your numerical algorithms

If you are comparing numerical results, e.g. when checking your test outputs, you should not check floating numbers for equality. There always may be round-off errors. Instead, check if the difference is lower than a specific threshold.

It is also a good idea checking your algorithms against different algorithms or formulating the scientific problem in a different way and comparing the results. If you get the same results using two or more independent ways, this is a good indication that your theory and your implementation is correct.

You could do those tests in your test code and use the fastest algorithm for your production code.

Answer 6

My advice would be to choose carefully how you expend your efforts. In my field (bioinformatics) the state of the art algorithms change so quickly that spending energy on error proofing your code might be better spent on the algorithm itself.

That said, what is valued is:

• is it the best method at the time, in terms of algorithm?
• how easy is to port to different compute platforms (different HPC environments, OS flavors etc)
• robustness - does it run on MY dataset?

Your instinct to build a bullet-proof codebase is noble, but it's worth remembering this is not a commercial product. Make it as portable as possible,error-proof (for your type of user), convenient for others to contribute, then focus on the algorithm itself