7

I have open source C# .NET project at GitHub with Appveyor CI + code coverage.

There are configurations like Release and Debug. There's also platforms like x86, x64 and Any CPU.

This results to following:

Configuration | Platform | Tests
--------------+----------+--------
Release       | x86      | 100/100
Release       | x64      | 100/100
Release       | Any CPU  | 100/100
Debug         | x86      | 100/100
Debug         | x64      | 100/100
Debug         | Any CPU  | 100/100

Should unit tests run always on all of the combinations (six) of these two dimensions when application is in development stage? ie. mainly only developers are in the process and lots of changes are taking place.

My intuition says that only Debug configuration and one platform should be tested.

My view is that Debug configuration gives more information and as C# being quite high level language and running in VM (CLR) there's no need to test x64 and x86 separately.

There's also third dimension with .NET framework version. And fourth with OS.

Is there best practises view in the field of what is or isn't necessary when application is being developed?

When shipping all tests are of course ran. Same with possible nightly/weekly/etc builds.

Improve this question
2
  • 2
    That's why most folks automate this. Configure your build procedure to build and run tests for as many different combinations as you have test mules to run them on. C# has 2 compiler optimization passes, so Release code can be very different than Debug. Debug asserts don't fire in Release builds.
    – Mark Benningfield
    Commented Jan 21, 2018 at 17:01
  • 1
    I’ve seen (cough written) code that only fails when built in Release mode. Makes it worthwhile to test against both IMO.
    – RubberDuck
    Commented Jan 22, 2018 at 16:39

3 Answers 3

Reset to default
7

This has nothing to do with "a common view in the field" - this is all about what makes sense most.

Running unit tests (or any kind of automated tests) using only one configuration and only one platform is fine as long as the code under test as well as the test itself don't show any indications for behaving differently under different configurations and platforms (except for performance, of course).

In a ideal world, there is only one configuration required for running the tests. However, certain kind of code constructs or statements are more prone to behave differently in "Debug" and "Release", or differently in "x86" or "x64". There are obvious things

  • like explicit tests #if DEBUG, and not so obvious things like that

  • subtle differences in floating point precision.

To find out what applies to your case, the only sensible way is, IMHO, to gain experience with the code base in stake. This might be done by inspection, or by occasionally running the tests using a different configuration. And it will take some time, depending of the size of the code base. In a large code base, you will probably find out that there are some parts or components where testing with one configuration is not enough, and others (hopefully the bigger part) do not need more than one configuration for testing.

If you ask which configuration you should prefer for unit testing, then it is IMHO obvious that

  • the configuration/platform used by the team in the code-compile-test-debug cycle needs to be the one for which you run the unit tests always

  • the configuration(s)/platforms used by your team for deployment should be tested before deployment, especially if you suspect they could behave differently.

You should also make sure everyone in the team is aware of the potential differences, and everyone who used some #if DEBUG statement is very careful about not changing any behaviour which could become relevant for testing.

Improve this answer
5

Unit tests are for verifying that you have the correct algorithm and that you haven't had a regression on a fixed bug. The first should generally be testable in debug mode, the second might not -- if you have an error that only occurs in a particular build (optimized release for instance, or different level of optimization), then obviously you need to retest it in that build.

Improve this answer
2
  • I corrected release to configuration. Not a software release as in version 1.0, but configuration without debug symbols, traces, etc enabled.
    – raspi
    Commented Jan 21, 2018 at 16:16
  • 3
    @raspi: Some problems can only be reproduced in (optimized) release builds. To verify that the problem is fixed (and later that you didn't re-introduce it), the tests for those problems need to be executed against a Release build.
    – Bart van Ingen Schenau
    Commented Jan 21, 2018 at 16:39
4

There are at least two things that can be different about Release, that suggests testing it.

  • Conditional compilation symbols may change, if any code is using that.

  • Optimization can alter the behavior, especially if there is a bug in the build system somewhere.

Minimally, you should test whatever build configuration you have chosen ship. If that's ANY CPU | Release, then minimally test that.

(There is also an additional complexity in that of the target environment. An ANY CPU could run differently on a 32-bit machine vs. 64-bit machine.)

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.