3

This is a question about terminology, not about method.

Our application consists of physical simulations. Results have been validated in various ways, but we have no oracle that guarantees correctness. Nonetheless, to improve and extend our software we need tests that ensure that at least we did not break things that worked satisfactorily so far.

Therefore we wrote tests that run a simulation and check whether the output, within a certain noise level, agrees with reference data that have been generated by running the very same simulation using the old, validated version of our software. How to call such tests?

A name that comes to mind is "regression test", but according to Wikipedia and other sources, this term is applied to whatever tests that have accumulated in a test suite; so it is not specific enough for our kind of tests.

6
  • from wikipedia Regression_testing : "Regression testing ... is re-running functional and non-functional tests to ensure that previously developed and tested software still performs after a change". Can you tell us where you get this from "according to Wikipedia and other sources, this term is applied to whatever tests that have accumulated in a test suite" ?
    – k3b
    Oct 19, 2021 at 7:50
  • "where you get this from"? isn't the logic the other way round?
    – Joachim W
    Oct 20, 2021 at 8:06
  • the first "..." is cited from en.wikipedia.org/wiki/Regression_testing
    – k3b
    Oct 20, 2021 at 15:52
  • This I know. I was rather concerned about the unclear wording "where you get this from" of yours. I still think that you mean it the other round: How do I get my "... whatever tests that have accumulated ..." from Wikipedia's "... tests to ensure that previously ... tested software still performs ..."? Is that your question?
    – Joachim W
    Oct 21, 2021 at 16:07
  • you said "but according to Wikipedia and other sources, this term is applied to whatever tests that have accumulated in a test suite; " My qestion: do you have a link for this?
    – k3b
    Oct 22, 2021 at 7:23

3 Answers 3

6

Characterization test is a term for this, introduced by Michael Feathers in Working Effectively With Legacy Code.

You're not testing whether the output is correct or not, and in many applications there is no precise definition of correctness. You're just using the test to guard against unintended changes to the behaviour of the subject under test.

18
  • I confirm, he introduces the term in Chapter 13 of his book (2005 edition).
    – Joachim W
    Oct 18, 2021 at 16:25
  • 1
    Had Feathers come here, maybe we could have suggested him a better term. Why not snapshot? Too late, his term has found some acceptance and already has a decent Wikipedia article, en.wikipedia.org/wiki/Characterization_test. So I will go for this. Thank you, @bdsl, and thanks to everybody for your helpful contributions.
    – Joachim W
    Oct 18, 2021 at 16:29
  • 1
    I would be quite confused if someone called the test the OP is describing a characterization test. The goal of a characterization test is to create a testing scaffolding around untested code to make it possible to change it safely by documenting how it currently behaves. This does not seem to be what the OP is using this test for. Oct 18, 2021 at 16:42
  • I commented at one point that I prefer this term over snapshot test. However, snapshot tests and characterization tests serve different purposes. In the book, "legacy code" is defined as code that doesn't have tests. A characterization test helps a developer understand what the legacy code does and refactor with confidence. Oct 18, 2021 at 16:45
  • 1
    @JoachimW So you are not testing against a previous version of the output, just what the output should be, is that correct? That is a very important distinction from your original question which you should definitely edit in. Which just makes your test... a test. Possibly a unit test, or an integration test, or maybe an end-to-end test, depending on how you test it. Oct 18, 2021 at 17:00
9

You're overthinking this. You're testing whether the output of the algorithm has regressed since the previous version of the code, so it's a regression test.

3
  • 1
    This is the fallback solution: We'd have a regression test suite that contains some regression tests in the narrower sense, and some other tests. Let's wait whether someone can help out with a more specific term.
    – Joachim W
    Oct 15, 2021 at 10:52
  • 1
    @JoachimW Still overthinking. It's a test that finds regressions, so call it a regression test. If somebody else says "that's not how we did regression tests at my company!" you can say "so?" Oct 15, 2021 at 12:28
  • 3
    @JoachimW Should we also not both call them a "test" because they're different? No, because they're both tests, just different kinds of tests. Should we call one a "regression" test but not the other? No, because they're both regression tests, just of different kinds of regression. Call it a "noise regression test" (or anything else that takes your fancy) if you need to disambiguate between the two, but this is much too niche to expect to find a standardized name that is conventionalized across the entire field of software engineering.
    – Flater
    Oct 15, 2021 at 12:32
2

I'm going to tackle your question from the standpoint of terminology rather than testing method, even though terminology and testing method can be intertwined. As you'll see further in my answer, this is all a matter of perspective.

The term "regression test" is very broad. While specific definitions vary, the essence of a regression test is to guard against problems appearing as the code base evolves. A regression test is not a methodology, or specific kind of test. The fact is, a unit test verifying a function returns false given certain input is also a regression test. So is an integration test that verifies an e-mail gets sent.

Basically, a "regression test" is something that should keep working as the code base is changed. This does not imply that a newly written test verifying existing behavior is somehow not a regression test. The time at which the test is written does not make it a regression test. The behavior the test verifies is what makes it a regression test. If that behavior is pre-existing, then a newly written test would simply be called "adding regression test coverage."

In a scientific application, we have plenty of tests that run a simulation and check whether the output, within a certain noise level, agrees with reference data that have been generated by running the very same simulation under an earlier version of our software.

This is just a rewording of the colloquial definition for regression test. It doesn't matter if you wrote a test this morning that verifies something that's been in the system for 10 years. It is still a regression test.

If we add "test methodology" to your question, then what you are talking about is still a regression test, but could also be another kind of test at the same time. Being that this is a scientific application, I assume the output to be quite complex, but still in the realm of raw data. In that case, Patrick McElhaney's answer also applies. Not only are you writing regression tests, but those tests could also be classified as snapshot tests. Arguably they could be called functional tests as well.

This happens frequently with testing terminology. There is no central authority defining these terms. How you classify a test largely depends on the context of the conversation. As a result, a single test can fall into multiple categories.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.