5

I had a bug that was really difficult to track down, because all the unit tests were green, but the production application didn't work properly.

Here's what happened:

I had a filter class that set my application to ignore data that was not in some specified time windows.

  • The unit test, which seemed thorough to me, turned green.
  • Additionally, my integration tests also produced results as expected.
  • Production, however, did not work.
    • As a result of the first two bullets, this problem was very difficult to find.

It turned out the problem was that my test dates were using my time zone (America/Chicago) but the production data was providing dates in UTC, which I did not realize, and the logic for the filter wasn't correct for UTC dates. (I was using joda time DateTime objects).

  1. Where did my workflow break down?

    • Did I fail to produce a spec that specified that the logic needed to handle dates in any time zone?
    • Did I fail to thoroughly consider all cases at the unit test level?
    • Did I fail to insure the integration test was sufficiently similar to production?
    • Other?
  2. What changes can I make to my workflow to better prevent this sort of mistake in the future?

  3. How can I more effectively debug a problem when there is an issue in production but not in testing?
  • 5
    The same way you deal with any other bug that surfaces: you fix the bug and write new tests that cover it. I think you have an expectation that your process is going to somehow produce a bug-free application; there is no such thing. – Robert Harvey May 27 '14 at 17:54
  • @RobertHarvey So what standard of thoroughness should one expect to achieve when writing a spec / unit test class? – durron597 May 27 '14 at 17:56
  • That the unit under test does what you expect it to do. – Robert Harvey May 27 '14 at 17:56
  • 3
    The one thing you can expect for sure is that you will never, never be able to think of all possible error scenarious. You will always overlook something. If only one thing is guranteed in SW development, it's this one. – JensG May 27 '14 at 17:59
  • 3
    re #2: always use UTC dates. Always. Local time is just for display – Steven A. Lowe May 27 '14 at 23:15
12

@RobertHarvey's comment is dead right: Process will not produce bug-free software, but good process can at least reduce the recurrence of classes of bugs. Sounds like you're attempting to accomplish the latter, so I will make my suggestion of where I think your mistake came in:

You failed to account for all the details of your production environment in your tests. Your tests should run in an environment that is as closely mimicing that of production as possible. In your case, you missed the fact that production deals with time in UTC while your unit tests were running with a different timezone.

Moving forward, just be aware of the way your production environment is configured and how it runs in as many details as possible. Whenever you go to write unit tests, look for things that may vary to effect your unit tests as possible, and try to make those variances match with what occurs in production.

That little forethought when writing your unit tests may help you moving forward catch other ways in which your tests don't run the same way as production does. Write it down in a checklist if you use those, either way just try to be mindful of it when writing your tests.

Process will not stop you from ending up with more bugs, even ones related to this one, but with that mindfulness moving forward hopefully the class of bugs known as "things that fail in production but not in development" will be reduced significantly.

One thing you may be missing that is common, is a document detailing the system that has specifics about all the things that vary. What timezone it runs on, what versions of languages it's written in, whether 64-bit or 32-bit, all of these things effect the behaviour of the system. Having them documented can help a team significantly to know more about the system than they often do individually. An individual developer frequently only knows subsections of the system, and may be unaware of these larger systemic details that effect them in ways they're otherwise unaware of.


  1. Where did my workflow break down?

    • Where your tests attempted to mimic production
  2. What changes can I make to my workflow to better prevent this sort of mistake in the future?

    • Document and or account for such variances between your dev and production environments.
  3. How can I more effectively debug a problem when there is an issue in production but not in testing?

    • Isolate the variables that may differ from system to system and check them between your dev and production environments. If you wish to learn it, using a debugger that can take a memory dump and let you analyze the live processes state can often help in this case, though learning that is far from an insignificant undertaking.
1

In addition to Jimmy Hoffa's answer: You ask wide net around all kind of bugs but your example is a bit more narrow: As far as I can see from your example you ran into an application weak-spot in that it doesn't deal well with localization. This is a big area where subtle bugs can come up and is very wide. Local differences and assumptions about those can give you headaches. Some examples:

  • Sunday is first day of the week in the US, Monday in Europe.
  • Month before day in US vs day before month in Europe.
  • But also using "," or "." when using values are different.

So my answer would be:

1) Where did my workflow break down?

It broke down because you didn't take localization into account.

2) What changes can I make to my workflow to better prevent this sort of mistake in the future?

Learn about localization and check you entire application for localization issues.

0

More to add to both: though process can't guarantee a bug free application, neither can 100% code coverage by unit tests.
In fact having that can very easily lead to a false sense of security, you wouldn't be the first team lulled into complacency by the idea that "we have 100% code coverage, now our system must be free of bugs".
Not only can there be bugs in the unit tests themselves, hiding bugs in the software, but unit tests do not test whether parts of the application work together as expected. If you have say a web service client and a web service, both unit tested, but against different specifications (say the specs changed after the service was written, the client was written to a different WSDL than what the service implements) your unit tests are never going to alert you about that.

You will also need proper integration testing, regression testing, functional testing (does the system actually implement the functionality required, if it doesn't it may be technically correct but it's still useless), usability testing (if it works, but is so kludgy the users don't use it or don't use it as intended, it's still a useless system). All are needed, all have their place, neither is on its own a guarantee that the system works correctly.

And of course all of them depend on properly written, well detailed, and well verified and signed off functional design documents, translated into correct technical design documents.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.