2

We have unit tests that are running some underlying model. We provide it with some test input, and receive some outputs + floating point scores. What's a good practice from a unit-testing standpoint? Should we place assertions on the floating points themselves, or just check that they are within a good range (i.e. > 0.5)? I am on the side to test everything as precisely as possible, but I do understand that a small model change can break all the tests. Maybe reg-testing / monitoring is a better way to catch model-output changes?

Thanks!

Improve this question
2
  • 1
    Recommended reading on StackOverflow: stackoverflow.com/questions/17333/… (while the question is tagged C++, it applies to floating point numbers in any language),
    – Ben Cottrell
    Commented Nov 5, 2021 at 17:48
  • 1
    I would say it depends on exactly what you are testing, which is not clear to me. Are you testing whether the output is correct, according to some defined behaviour or are you just testing whether some abstract goodness score of the model is the same as it was before? In other words: How do you know what is the right result and how much accuracy do you require from the results?
    – Helena
    Commented Nov 5, 2021 at 22:49

1 Answer 1

Reset to default
3

If the result of the test is deterministic (and if it isn't, why not), then you should absolutely compare the actual value you received from your model to the expected output. Given that they are floating point numbers, though, you don't want to do a simple equals comparison because of how floats are stored. Rather, ensure the two values are within some small tolerance of each other, e.g., Assert.IsTrue(Math.Abs(expectedValue - observedValue) < 1e-3);

The tolerance you use can depend on the scale of the values as well as your problem domain. For example, if you're dealing with values on the order of 1e-3, your tolerance may be 1e-6. If you're dealing with critical care systems, your tolerance may be even smaller.

If changes to your model are expected to induce large (> 1e-3) changes to the results, you'll need to modify the expected values in your tests at the same time - and ensure that those changes are expected and correct.

Improve this answer
5
  • If at all possible I would avoid asserting on the truth of a condition, because then the only diagnostics you get are "expected true/false to be false/true". Ideally the output should show you the value you actually got.
    – jonrsharpe
    Commented Nov 5, 2021 at 19:10
  • I’d make the tolerance as small as possible while still passing the test. If you exceed a tolerance of 0.0001 (ten times smaller the you said), either you have numerical problems that need fixing, or you have a bug that needs fixing as well, or you need to switch from floor to double.
    – gnasher729
    Commented Nov 5, 2021 at 20:02
  • 2
    The main problem IMHO is that floating point numbers are used naively most of the time despite the fact that they can get quite nasty under the surface. This question is exactly pointing to one of those problems: missing requirements! You need to specify the accuracy of the results of a floating point calculation. This can either be done externally by the customer of your result (e.g. position accuracy shall be < 0.3 meters), or internally by analysis of the algorithm at hand. Errors in the algorithm are then found when the results are too inaccurate.
    – Hartmut Braun
    Commented Nov 6, 2021 at 10:06
  • For Python specifically (which is a tag for the question), the unittest module has an assertAlmostEqual method exaclty for this purpose: docs.python.org/3/library/…
    – jfaccioni
    Commented Nov 9, 2021 at 13:41
  • How to unit test two lists with floating values?
    – Devharsh Trivedi
    Commented Mar 14, 2023 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.