18

I would like to ship test code alongside a product. Specifically, provide an option so that anyone with a copy of our program can hit a "self-test" button or pass --self-test on the command line and run through the complete suite of unit | integration tests.

I mostly want to do this to help debug problems discovered in the field, so when a bug report is incoming from an end user there's a chance of it being supported by "and also, these three tests failed on my machine". I'd like manual testers to be able to run the unit | integration tests as well.

However, the test of the team believes that test code is not production code and therefore shall not be shipped. I don't really get this argument, since most open source projects ship a test suite. It does seem to be unusual in closed software.

I would like supporting evidence or anecdotes for either side of the argument. I've taken my best guess at which stack exchange site would be most appropriate, but please let me know if this is out of place.

  • 8
    Why would a unit test in a closed source program (or an open source program that hasn't been modified) ever fail? If your product requires a fair amount of setup and setup issues are often the source of bugs, it might make sense to ship some sort of "validate my configuration" app that does things like validating the database connection, validating connections to any other external services your code depends on, etc. It wouldn't make sense for a unit test to ever fail, though, since you've already validated that the code works. – Justin Cave Jun 21 '15 at 22:14
  • recommended reading: How do I explain ${something} to ${someone}? – gnat Jun 21 '15 at 22:16
  • 15
    Why would a unit test fail in the field? Off the top of my head: Corrupt program. Dodgy hardware. Race conditions we didn't see locally. Linked to a different dynamic library. Conflicts with antivirus or OS. Being used with surprising version of related software due to incomplete update. Interaction with other processes not behaving as expected. There's loads of reasons why bugs turn up in the field, and a lot of them could be hit from a unit test (for a given definition of unit) – Jon Chesterfield Jun 21 '15 at 22:34
  • 7
    @JonChesterfield: creating a self-test feature in your program is probably a good thing. And if that self-test feature can partly reuse code from your unit tests, why not? But such a feature as well as the reusable parts should be developed with a "it is production code" point-of-view. – Doc Brown Jun 22 '15 at 8:28
  • 2
    @JonChesterfield I have a hard time picturing a unit test failing on most of those causes. Integration tests are another matter, though--I can see merit in shipping them if it can be done without too much extra stuff. – Loren Pechtel Jan 3 '16 at 4:12
19

Sometimes test code contains snippets of code from third parties, both external and internal to your company. This happens as users file bugs; your tests (such as regression tests) then incorporate the code supplied by them to reproduce. Often, the licensing of such code snippets to reproduce bugs is unclear. So, you should be aware of intellectual property issues. You don't want to ship test code that accidentally reveals some trade secrets or intellectual property of a different department of your company, or of your external partners either.

On another note, test code is rarely held to production code standards: code reviews aren't necessarily done; coding standards not enforced, etc.. This is unfortunate, yet commonplace, and should not necessarily a reflect poorly on the test team if they didn't have that goal in place at the time these tests were developed.

On the other hand, many tests are simply embarrassingly bad, and don't even test what some think is being test. That is different issue...

Ultimately, because of all these factors, you might want to classify your tests as those that can be shipped as open source, and those that simply can't. (You may want to write some custom tests with shipping them in mind, slowly migrating the others into that set.)

  • The third party issue is a really good point. Grouping the test code into "externally visible" and "maybe confidential" would be error prone and a considerable overhead. That's pretty much a deal breaker all by itself, thank you. – Jon Chesterfield Jun 21 '15 at 22:41
  • Yes, hard to do after the fact. I would think you'd have more luck with a dedicated effort to develop shipping tests. – Erik Eidt Jun 21 '15 at 22:47
  • @ErikEidt: I took the freedom to make a suggestion for removing "as open source", because that is probably not what the OP had in mind - I think he wants to ship the tests as closed source. – Doc Brown Jun 22 '15 at 8:24
  • @DocBrown, I take your point. Probably a matter of interpretation as the OP did mention "open source" somewhere in the post. In any case your edit generalizes the point nicely. – Erik Eidt Jun 22 '15 at 17:05
18

Shipping tests? Yes. Shipping Unit tests? No.

As you say in the comment, problem you may face when running the product on a client computer will include problems such as linking with the wrong dll, generally this is not something a unit test will catch (which will no doubt have mocked the dll out to test the code).

Now, shipping an integration test suite, that calls the UI that calls the logic that calls the dll... that will work much better. Integration tests can show other aspects of failed installations that unit tests would not show up. (eg my current product requires an installation of k-lite codecs, which we are not allowed to bundle due to licensing. Unit tests may show our code working fine, but still not working to the customers satisfaction. Similarly, our configuration of the codecs may not have worked correctly, unit tests would also not show that up).

So - ship some of your integration tests instead, which would be exactly what you want for a installed, integrated, product.

2

I could understand this concern strongly in areas where you are covering every single inch of the hardware, like a multithreaded next-gen AAA game engine which uses every single CPU core, SIMD intrinsics, GPU, GPGPU, etc. while delivering a cross-platform product.

In those cases, your worst nightmare will often be those cases where your tests (unit and integration) will pass for the first 5,000 disparate machines/platforms tested, but fails for the 5,001th due to a driver bug for an obscure GPU model Just thinking about this gives me the shivers -- you cannot possibly test or foresee these in advance.

Especially if you write GPU shaders, you can end up playing a reverse lottery where half the code you write will invoke undefined behavior, since there are few portable standard guarantees enforced by all GPU models/drivers involved. While it's getting less and less like playing minesweeper these days, this should give people some idea: http://theorangeduck.com/page/writing-portable-opengl. Trying this in the late 90s and early 2000s was really horrible, and it was minesweeper all the way.

For these kinds of cases, you often need teams of 10,000+ testers with a really wide range of hardware and operating systems to really solidify the product and feel confident about it prior to a stable release. Not all companies can afford to have such a wide test base, and not all have the discipline to do it right (all widely-noticeable issues should be fixed prior to having so many testers in some internal pre-alpha/alpha stage or else the flood of redundant reports can throw developers into a patch-and-pray panic).

What I recommend in this case is what others suggested, focus on a distributed set of integration tests. You can bundle it up with the installer, requiring users to pass a basic diagnostics check with careful attention to providing details as to why the installation failed that they can pass to you, the developers.

Another thing (if you can convince the boss) is to have a wide range of hardware available to do contiguous integration. The more variety in hardware/OS combos, the merrier. You want even a variety of crap hardware that models the bare minimum hardware requirements for your CI servers: you never know.

But there's one more thing I'd suggest:

Logging

If you are dealing with anything like the scenario I described above, then often you cannot possibly test for these things which tend to be the most problematic (those worst possible gotchas which show up at the worst possible time and cannot possibly show up in even the most exhaustive test suite since it's an issue constrained to a very specific hardware/OS combo).

Yet most of those kinds of issues like obscure hardware incompatibilities or outright driver glitches or linking against the wrong dylib (I've never actually faced this concern) won't get you far past starting up the software. It's typically gonna crash and burn pretty soon, crudely speaking.

I recommend, for sanity sake, to kind of embrace the inevitable. You can't possibly do anything about these things you cannot possibly test comprehensively. Don't try to prevent the hurricane (impossible), but board up those windows.

Typically here, the best thing we can do is find out the problem as soon as possible, where it occurs as detailed as possible (to narrow our list of suspects), and have the issue fixed ASAP after it's reported.

In this case, logging can be a lifesaver. For these kinds of fields, you can create these spammy technical logs which no one would ever read through. Often relevant is just the very last line recorded in the log before the user faced a crash due to a driver glitch, e.g. You can write an external process or hook which monitors for crashes and then shows the last line of the log that users can copy and paste to you, e.g. in addition to a crash dump.

Since this often needs granular information and a lot of the most susceptible areas in code to these hardware/platform/driver issues is performance-critical, there's this awkward issue where the logging can be happening at such a frequent rate that it'll actually slow down the software.

A useful trick in this case is to rely on the assumption that something executed once will execute successfully the second time, third time, etc. This is not the most sound assumption, but it's often "good enough" (and infinitely better than nothing). With that, you can use a little bit of external state to keep track of when something has been logged already and skip subsequent attempts to log for those really granular cases where the code will be invoked repeatedly in a loop.

Anyway, I hope this helps. I've run into this kind of temptation in the past and have a bit of a paranoia surrounding GPU coding (GPGPU and shaders) as a result of some past experiences among myself and my team (sometimes just seeing other team members deal with these really late and post-release gave me the creeps, like some ATI glitch on a specific Radeon model which would crash on rendering antialiased lines, later reported and marked as a known problem with only a workaround solution available).

Logging was the thing that saved our butts there, letting us often see the issue on that 10,001th obscure prototype machine with an onboard GPU we never heard of, with the last line of code immediately letting us spot exactly where the failure was down to 2 or 3 lines of code as suspect, e.g. If it's inside an elaborate shader, we're kind of SOL since we can't do logging inside a GPU shader, but we can at least use logging to see which shader had the issue right away to start off the investigation.

  • 2
    Memoizing logging code is clever. We don't currently produce logs - largely because of performance concerns - so debugging involves a core dump. Embedding diagnostic tests with the installer is a good idea too. Thank you for the detailed reply. – Jon Chesterfield Jan 2 '16 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.