Does software which implement scientific models require unit tests?

Question

I work in a field where lots of code is written, but hardly ever tested. This is because we are foremost scientists who try to solve problems with code. The few coding courses we had, focused on the basics and many have never heard of git, unit testing, clean code after graduating. Many haven't even heard of those during their PhD...

Maybe it's better now, but 10-5 years ago we did not have any mandatory courses which cover those areas.

Often the software solves differential equations numerically. In many cases PDEs with many feedbacks going on.

Think of weather predictions, chemical reactions, atmospheric models and so on.

So now my questions, would you trust results of a complex software with many hundreds or thousands of functions without a single unit test? If there are tests then they are rather high level, like to check if the results stay the same with the same input or if the results of a very simple case fit an analytical solution.

Even if you know that the numerical solution of the equation is sound, based on some years old publication, would you trust the model to make predictions? Would you trust it if it can cause billions of damage or even loss of lives?

On a side note, often these models are compared against each other with the same simplified inputs.

Answer 1

A few aspects I would like to touch on.

I work in a field where lots of code is written, but hardly ever tested. This is because we are foremost scientists who try to solve problems with code

I think this is common in science. And I think it's only partly due to lack of courses or motivation.

I think the main reason is that a lot of scientific code is more prototyping than application development. A lot of it is used for a few analyses and abandoned. It's small, so you can test by hand.

One of the main benefits of unit tests is for long-term maintenance and refactoring. If your code won't be maintained long, and you won't refactor it, it's reasonable to prioritize unit tests less.

But a part of the software is reused a lot (unfortunately not usually clear beforehand). And then...

Would you trust it if it can cause billions of damage of even loss of life?

At this point we've left 'prototyping' and entered application development. I'd assume the code is maintained a long time by multiple people. It'll likely be refactored if it keeps growing. It has probably long ago stopped being possible to test everything by hand for most changes.

And, of course, risk tolerance would be much lower if the possible damage is greater.

Unit tests become much more valuable due to all that. I think it pays to follow better software engineering principles like unit testing at this point, and honestly a while before this point.

Often the software solves differential equations numerically. In many cases PDEs with many feedbacks going on.

I think the more important quality is scale (lifetime, collaboration, change frequency, complexity...), not so much whether there are scientific models.

But I'll say that such things are actually quite easy to test automatically (whether or not you'd still call it a 'unit' test). No UI or external dependencies to be mocked.

The more examples and edge cases are covered, the more one would trust it. It probably takes some scientific insight into how 'well behaved' the model, and knowledge of the risks, to know how much is enough.

often these models are compared against each other with the same simplified inputs.

That would actually give me quiet a bit of confidence. I think it's a good method of validation and bug detection.

It doesn't help much with localizing problems though - you might not even know which of the models is wrong, let alone what is wrong with it. Unit tests could help with that.

Answer 2

It's something you can actually test scientifically. You don't have to rely on arguments from the Internet. Write unit tests and see if they catch errors your manual testing didn't. See if they reduce the time to find errors.

Unit testing wasn't very common in software development until the early 2000's, so anyone who has been doing this for longer than around 15-20 years will remember what it was like without it. As one of those people, I can tell you I wouldn't trust software without unit tests unless you are literally spending weeks checking for bugs every time you make a change.

Answer 3

More people are thinking that research software should see some standardized testing. One of the problems with spending time writing quality software in science is getting recognition for it in a culture where papers are the currency. The Society for Research Software Engineering is trying to change that for everyone's benefit.

Last century, your safety net was depending on extremely well-tested libraries, such as the NAG libraries for Fortran and Numerical Recipes (Fortran/Pascal/C), for your serious computations. That, and having a post-doc/grad student whose job it was to get the right numbers. :)

Answer 4

would you trust results of a complex software with many hundreds or thousands of functions without a single unit test?

I would not.

But a properly written set of unit tests is only one side of it.

Unit tests should be complemented by black box end-to-end tests that cover major functionality.

would you trust the model to make predictions?

Now when you know how important to test your code, you should be able to separate the model from its implementation and answer this questions: "I'm confident my implementation is correct, so the model must be a dud".

That's why, BTW, when software might cause serious damage or loss of life there are special engineering practices developed for it, like two independent implementations of the same thing.

Answer 5

Rigorous testing is not synonymous with unit testing. Yes the software should be rigorously tested; but no, not necessarily unit tested.

Answer 6

As the story goes accoring to Uncle Bob (you can read it here currently), in the 1950s-60s, the programmers who wrote the code for the Mercury space capsule wrote their unit tests in the morning and made them pass in the afternoon.

If lives and billions of dollars are involved, it is just common sense do to rigorous testing. That being said, if rigorous testing was done manually at first, then later on detailed regression tests may be enough to ensure that the code keeps working.

Answer 7

It is important to test against regression. It is easy to reintroduce an error or bug that was solved earlier. The time you fix something, you require a unit test to be written for it. Some of the bugs are not even in your control as you might depend on third party libraries. To fix a failed unit test, it might be as simple as reverting to an earlier version of a library. Bugs might introduce themselves with a failure, or simply render the wrong result. The latter can be hard to catch without some unit tests.

Answer 8

Rigorous software testing is not common in physical sciences. This issue caused somewhat of an existential crisis in the scientific computing community starting in the 90's about how reproducible a study can be if the methods are not thoroughly vetted. Most of the effort on addressing this has gone into standards for journals to at least require disclosure of source code to reviewers, and in many cases to require that code be made public and permanently archived.

Relatively little attention has been paid to validating code itself. Any particular research code is frequently used only once for a particular paper. To some extent this question can be addressed by the basic fact that reproducing results often means re-implementing functionality from scratch, and it should become clear if two codes unexpectedly produced different results. This is obviously far from perfect, but it seems to be the general attitude towards this problem at least within my discipline (geophysics).

General community attitude aside, to address the question itself- Does scientific software require unit tests? I think the answer really comes down to the context and scope of the code in question.

The term "unit test" is a problem here. The concept of unit testing comes from development of software libraries, in which the code base provides many entry points which can be isolated to a large degree from one another. Tests are usually implemented as independent source files, each with a main() function that makes a single api call to the library, and checks that the result is as expected. That API function may rely on other functions, and may require some amount of resource mocking, but it's still a self-contained "unit" with clearly defined known inputs and outputs.

Scientific code rarely works that way. It usually presents a single entry point to the user which reads a huge set of parameters as input, either through an input file, CLI flags, or GUI. Any particular set of input parameters may cause the code to only touch a small subset of the functionality in the application, and the resulting output is frequently (by definition for research software) difficult to predict apriori.

Typically then, it makes most sense to do some form of "benchmarking." Note that this is not the computer science variety of benchmarking which focuses on efficiency. In physical science context, benchmarking usually refers to reproducing some previously known solution, and comparing the code's outputs with what's expected. In cases where no analytic solution exists this might be a comparison to an analog experiment, or just output from some other similar code. It is becoming increasingly common for reviewers to expect some form of benchmark comparisons, especially in cases where the results are particularly surprising or anomalous.

Scientific software that gets frequently reused might undergo more thorough testing, but true unit tests are still uncommon for the reasons mentioned above. Rather, an increasingly common approach is to track suites of input parameters and output values in a similar style to unit test suites. Whether the results are correct is not necessarily addressed, but it's easier to gain confidence that bugs have not been introduced as the software gains complexity.

I'm aware of one paper about this technique, but there are likely others out there. https://arxiv.org/pdf/1508.07231.pdf

Answer 9

I worked not in the hard sciences, but in the business department, and there are a lot of people using simulation research (simulated economic decision-making, etc). I myself published some simulation research, and in fact there was an error in my code that led to incorrect results and graphs being published. I found out much later, and there was no interest from the publisher in updating the online version of the paper or posting an "errata".

There is simply no incentive in academia to do this kind of disciplined software development. It is part of the problem of "peer review" that reviewers can only really read the paper and say "that sounds like you've done it right" or not. It's not customary to share research data or code with reviewers, and even if you did, they're not going to duplicate your very hard work to re-run your analyses or re-implement your models to make sure you did it right.

How would we fix it? I think the pressure is not likely to come from journal or conference editors, as they share the incentive that all researchers do: more publications mean higher rankings mean jobs, tenure, and promotion. Instead, maybe the pressure could come from the NSF, the DOD, or other agencies that provide grant funding. To them, it would be of value to have stronger validation that data and code do what they're supposed to do. Institutions then would have to invest money in having actual skilled software developers support researchers -- researchers themselves have enough to keep track of that they can't spend the time to get good at coding at the same time.

Answer 10

It depends what you use unit tests for. Your software can be in a state where you say “if any if the unit tests failed, the software just couldn’t work.

In this case unit tests help you just during refactoring. You make mistakes. Trivial mistakes the compiler doesn’t compile. Slightly less trivial you get a warning. Even less trivial you brew a unit test. Anyway the unit tests just prevent trivial errors and are useless until you refactor again.

Other unit tests solve a complex problem and compare against a known correct solution. Thats a lot more work, might be an integration test, and is definitely useful in scientific software.