27

I work in a field where lots of code is written, but hardly ever tested. This is because we are foremost scientists who try to solve problems with code. The few coding courses we had, focused on the basics and many have never heard of git, unit testing, clean code after graduating. Many haven't even heard of those during their PhD...

Maybe its better now, but 10-5 years ago we did not have any mandatory courses which cover those areas.

Often the software solves differential equations numerically. In many cases PDEs with many feedbacks going on.

Think of weather predictions, chemical reactions, atmospheric models and so on.

So now my questions, would you trust results of a complex software with many hundreds or thousands of functions without a single unit test? If there are tests then they are rather high level, like to check if the results stay the same with the same input or if the results of a very simple case fit an analytical solution.

Even if you know that the numerical solution of the equation is sound, based on some years old publication, would you trust the model to make predictions? Would you trust it if it can cause billions of damage of even loss of live?

On a side note, often these models are compared against each other with the same simplified inputs.

Improve this question
11
  • 3
    Does this answer your question? Unit testing for a scientific computing library
    – gnat
    Aug 5, 2020 at 17:12
  • thanks, but not really, I know how to do it in general, but I would like the view of experienced software developers if unit tests are necessary in order to trust the results, or just nice to have during development.
    – gogoolplex
    Aug 5, 2020 at 17:19
  • 4
    Scientific models don't require tests, but they certainly benefit from them. Tests help ensure that you don't break things when making changes. The last modeling tool I worked with (product demand and pricing) had many tests. Breaking changes might not result in errors, but rather invalid data that could result in dramatic financial losses.
    – Dan Wilson
    Aug 5, 2020 at 18:45
  • 4
    Also, hire a seasoned software developer to work in the same team as the scientists if you can. He/she will teach you the best practices while working on getting your code and deployment (if any) together. It may also be good for recruitment, since the developer can learn a bit into your scientific field "for free" while working on a paid job. Such an opportunity often attracts great minds.
    – fviktor
    Aug 19, 2020 at 14:19
  • 2
    Publication is key here, as peer review is how scientific results gain credibility. Having reproducible results is getting more attention/oversight in peer review and some more software focused journals, e.g. JOSS, require, at minimum 'documented manual steps' to objectively demonstrate functionality.
    – jbosq
    Aug 19, 2020 at 17:35

8 Answers 8

Reset to default
33

A few aspects I would like to touch on.

I work in a field where lots of code is written, but hardly ever tested. This is because we are foremost scientists who try to solve problems with code

I think this is common in science. And I think it's only partly due to lack of courses or motivation.

I think the main reason is that a lot of scientific code is more prototyping than application development. A lot of it is used for a few analyses and abandoned. It's small, so you can test by hand.

One of the main benefits of unit tests is for long-term maintenance and refactoring. If your code won't be maintained long, and you won't refactor it, it's reasonable to prioritize unit tests less.

But a part of the software is reused a lot (unfortunately not usually clear beforehand). And then...

Would you trust it if it can cause billions of damage of even loss of live?

At this point we've left 'prototyping' and entered application development. I'd assume the code is maintained a long time by multiple people. It'll likely be refactored if it keeps growing. It has probably long ago stopped being possible to test everything by hand for most changes.

And, of course, risk tolerance would be much lower if the possible damage is greater.

Unit tests become much more valuable due to all that. I think it pays to follow better software engineering principles like unit testing at this point, and honestly a while before this point.

Often the software solves differential equations numerically. In many cases PDEs with many feedbacks going on.

I think the more important quality is scale (lifetime, collaboration, change frequency, complexity...), not so much whether there are scientific models.

But I'll say that such things are actually quite easy to test automatically (whether or not you'd still call it a 'unit' test). No UI or external dependencies to be mocked.

The more examples and edge cases are covered, the more one would trust it. It probably takes some scientific insight into how 'well behaved' the model, and knowledge of the risks, to know how much is enough.

often these models are compared against each other with the same simplified inputs.

That would actually give me quiet a bit of confidence. I think it's a good method of validation and bug detection.

It doesn't help much with localizing problems though - you might not even know which of the models is wrong, let alone what is wrong with it. Unit tests could help with that.

Improve this answer
1
  • This is a great answer, I would add that unlike commercial code, scientific code is often used by the programmer themselves, and under much narrower applications. Both of these things reduce the advantages of programmatic testing.
    – taciteloquence
    Aug 20, 2020 at 5:33
11

It's something you can actually test scientifically. You don't have to rely on arguments from the Internet. Write unit tests and see if they catch errors your manual testing didn't. See if they reduce the time to find errors.

Unit testing wasn't very common in software development until the early 2000's, so anyone who has been doing this for longer than around 15-20 years will remember what it was like without it. As one of those people, I can tell you I wouldn't trust software without unit tests unless you are literally spending weeks checking for bugs every time you make a change.

Improve this answer
5
  • +1 And if possible, have somebody else write the unit tests / generate the test data and expected results. Speaking from experience, it can be easy to introduce the same error into both your algorithm and your test data. Having two people do each of those tasks separately reduces that chance.
    – mmathis
    Aug 5, 2020 at 18:16
  • Thanks for the answer. Yes that's part of the problem, many of those larger models have been developed since the 90s, with many developers which haven't heard about unit testing. I asked because I am thinking about making an effort and introduce unit testing to the community.
    – gogoolplex
    Aug 5, 2020 at 18:24
  • @mmathis Wouldn't that be acceptance tests? Unit tests are generally about checking that each component of the software works as the developer intended, which is distinct from checking that the software as a whole produces the right results. In innovative simulation software you might not even have any way to generate the expected results without duplicating the effort to build a simulation.
    – bdsl
    Aug 5, 2020 at 18:55
  • @bdsl Whatever you call them, or whatever kind of test they actually are, it helps to have somebody else calculate the expected results. I'm not advocating the OP write a test to verify the complete weather simulation produces some set of expected results given an input, but there are many scientific / mathematical models / algorithms where this is the case - a method to integrate PDEs, for example, or to compute a Fourier Transform.
    – mmathis
    Aug 5, 2020 at 19:19
  • @mmathis Right, I think there's value in having both types of test. Acceptance tests it makes to have written by someone else. Having someone else write unit tests probably slows things down too much unless you work as a pair simultaneously. Having someone else review unit tests is absolutely worthwhile.
    – bdsl
    Aug 5, 2020 at 19:25
5

More people are thinking that research software should see some standardized testing. One of the problems with spending time writing quality software in science is getting recognition for it in a culture where papers are the currency. The Society for Research Software Engineering is trying to change that for everyone's benefit.

Last century, your safety net was depending on extremely well-tested libraries, such as the NAG libraries for Fortran and Numerical Recipes (Fortran/Pascal/C), for your serious computations. That, and having a post-doc/grad student whose job it was to get the right numbers. :)

Improve this answer
3

would you trust results of a complex software with many hundreds or thousands of functions without a single unit test?

I would not.

But a properly written set of unit tests is only one side of it.

Unit tests should be complemented by black box end-to-end tests that cover major functionality.

would you trust the model to make predictions?

Now when you know how important to test your code, you should be able to separate the model from its implementation and answer this questions: "I'm confident my implementation is correct, so the model must be a dud".

That's why, BTW, when software might cause serious damage or loss of life there are special engineering practices developed for it, like two independent implementations of the same thing.

Improve this answer
1
  • +1 for mentioning black-box end-to-end tests. For some code, writing unit tests can be expensive and accomplish less than end-to-end tests.
    – user1071847
    Aug 20, 2020 at 14:34
3

Rigorous testing is not synonymous with unit testing. Yes the software should be rigorously tested; but no, not necessarily unit tested.

Improve this answer
2

As the story goes accoring to Uncle Bob (you can read it here currently), in the 1950s-60s, the programmers who wrote the code for the Mercury space capsule wrote their unit tests in the morning and made them pass in the afternoon.

If lives and billions of dollars are involved, it is just common sense do to rigorous testing. That being said, if rigorous testing was done manually at first, then later on detailed regression tests may be enough to ensure that the code keeps working.

Improve this answer
1

It is important to test against regression. It is easy to reintroduce an error or bug that was solved earlier. The time you fix something, you require a unit test to be written for it. Some of the bugs are not even in your control as you might depend on third party libraries. To fix a failed unit test, it might be as simple as reverting to an earlier version of a library. Bugs might introduce themselves with a failure, or simply render the wrong result. The latter can be hard to catch without some unit tests.

Improve this answer
1

Rigorous software testing is not common in physical sciences. This issue caused somewhat of an existential crisis in the scientific computing community starting in the 90's about how reproducible a study can be if the methods are not thoroughly vetted. Most of the effort on addressing this has gone into standards for journals to at least require disclosure of source code to reviewers, and in many cases to require that code be made public and permanently archived.

Relatively little attention has been paid to validating code itself. Any particular research code is frequently used only once for a particular paper. To some extent this question can be addressed by the basic fact that reproducing results often means re-implementing functionality from scratch, and it should become clear if two codes unexpectedly produced different results. This is obviously far from perfect, but it seems to be the general attitude towards this problem at least within my discipline (geophysics).

General community attitude aside, to address the question itself- Does scientific software require unit tests? I think the answer really comes down to the context and scope of the code in question.

The term "unit test" is a problem here. The concept of unit testing comes from development of software libraries, in which the code base provides many entry points which can be isolated to a large degree from one another. Tests are usually implemented as independent source files, each with a main() function that makes a single api call to the library, and checks that the result is as expected. That API function may rely on other functions, and may require some amount of resource mocking, but it's still a self-contained "unit" with clearly defined known inputs and outputs.

Scientific code rarely works that way. It usually presents a single entry point to the user which reads a huge set of parameters as input, either through an input file, CLI flags, or GUI. Any particular set of input parameters may cause the code to only touch a small subset of the functionality in the application, and the resulting output is frequently (by definition for research software) difficult to predict apriori.

Typically then, it makes most sense to do some form of "benchmarking." Note that this is not the computer science variety of benchmarking which focuses on efficiency. In physical science context, benchmarking usually refers to reproducing some previously known solution, and comparing the code's outputs with what's expected. In cases where no analytic solution exists this might be a comparison to an analog experiment, or just output from some other similar code. It is becoming increasingly common for reviewers to expect some form of benchmark comparisons, especially in cases where the results are particularly surprising or anomalous.

Scientific software that gets frequently reused might undergo more thorough testing, but true unit tests are still uncommon for the reasons mentioned above. Rather, an increasingly common approach is to track suites of input parameters and output values in a similar style to unit test suites. Whether the results are correct is not necessarily addressed, but it's easier to gain confidence that bugs have not been introduced as the software gains complexity.

I'm aware of one paper about this technique, but there are likely others out there. https://arxiv.org/pdf/1508.07231.pdf

Improve this answer
2
  • Thanks for your input. Yes you might have a single entry point, but your program should still consist of smaller parts like, modules, functions, classes. If, not then you have another problem. Unit testing might actually also force you to write smaller chunks of code with no side effects. I think "benchmarking" should complement unit tests.
    – gogoolplex
    Oct 4, 2020 at 6:49
  • Yes, but the problem is that scientific code uses libraries for even quite high level tasks, to the point that the smallest scale modules in the actual code often do "simple" things like calculating a coefficient based on state variables, which rely on other state variables and so on. There is some level in any code base at which the pieces are no longer independent of one another, and that's frequently the starting point for scientific software because lower level tasks are taken care of by libraries (which should have unit tests, of course).
    – jmp
    Oct 4, 2020 at 18:43

Not the answer you're looking for? Browse other questions tagged or ask your own question.