Unit tests became unstable after upgrading compilers and IDEs

Question

I have a set of about 300 unit tests that have been through a difficult few months. The poor tests were subject to being upgraded from a V110 compiler (Visual Studio 2008) alongside Visual Studio 2012 to a V120 (Visual Studio 2013) compiler alongside an upgrade to Visual Studio 2015.

These tests were passing in VS2012. They broke horribly after the compiler/IDE upgrade. Some things were very odd, I spent weeks trying to fix them, worked with several engineers, and found very little traction. Then a beam of hope came in the form of VS2015 Update 3. It fixed the tests. For about 2 days. Now I'm met with a dreaded, "Failed to set up the execution context to run the test", for a test that was working last week.

So, the tests have been a bit battered recently. These are critical tests that my team needs to be running (and that we can't put in the gated builds because of the failing tests). We have workarounds to run the tests piece-by-piece, in an older IDE, from the command line, or any other number of tricks and hacks...but we really can't have tricks and hacks for important unit tests.

I'm not asking how to fix this specific set of tests, but moreover how should I as a software engineer start to regain stability with a set of unit tests that became unstable. As I'm writing this the answer seems obvious: Snap off the smallest subset of failing tests and analyze the failures/start the debugging process (which of course I'm doing), but things always have a rub. The upgrade happened, and there's no going back, so at the moment we're stuck with some (occasionally) unreliable tests.

Because the tests can be (not easily and not always all of them) ran, fixing these tests doesn't ever seem to be a higher priority for my team than pushing out new features. I'm an SD1 and so I don't have a lot of sway to get the more senior members of the team to analyze these test failures and I feel like I'm spinning my wheels trying to fix tests while waiting for the next test/tool failure. What should I do?

Answer 1

The problem you describe is in no way restricted to unit tests, and when approaching this from a software engineering perspective, one might ask the more general question

"We picked a certain framework as a platform for our program system, but when upgrading to a newer version, we run into lots of unexpected compatibility problems, because the vendor did not make a good job in respect to downwards compatibility. What can we do?"

The possible answers to this question might not please you, but the measures are well known:

• make sure your software is not more tightly coupled to the framework than absolutely necessary

• make sure whenever you install an upgrade of the framework, there is a way back to the former version when this causes too much problems

• make sure you know your framework well, and avoid to use the "latest and greatest features immediately", better be a little bit conservative, rely on mature features

• when the problems become intolerable, consider to change your platform. Try to pick a framework which has a reputation to be mature and stable, especially when it comes to backwards compatibility.

In your case, for example, you used the unit testing framework of Visual Studio. Of course, frameworks often force you to couple your software to them tightly, but in this case, whenever you need to ugrade the IDE, you also need to upgrade the unit testing environment. AFAIK you cannot upgrade those components individually. As you wrote, there is currently no easy way back for you to a former version of VS.

So is there an alternative? Yes there is - pick a different unit testing framework which is not so tightly coupled to the IDE. Unit tests directly inside the IDE is a nice gimmick, but nothing you cannot live without.

For example, we have a full unit testing suite for one of our products since more than ten years, developed with NUnit, and we never encountered any of the problems you mentioned when we upgraded from VS 2003 to 2015 with almost any intermediate VS version which was released by Microsoft. NUnit was available at a time where VS unit testing was not, and nowadays there are VS plugins to run NUnit tests directly from inside the IDE. Nevertheless, the latter is still dispensable, we can also run all tests outside VS, just by using the NUnit GUI.

Of course, when we upgraded the NUnit version from time to time, we had to deal with some minor issues, but never anything which was as severe as you described it, not even close.

There might be other Unit testing frameworks with similar properties and also not so entangled with the IDE; the correct choice needs surely an evaluation for your case. And when you pick such a framework, you have an additional third party vendor on whom you have to rely on, which can have a lot of new drawbacks and dependencies, too. Maybe our team was just lucky to pick "the better" framework several years ago. If one is not so lucky, and picks some framework which is abandoned after some months by the vendor - well, shit happens.

But I think this is the best recommendation I can give you here. Maybe changing the framework is not an option for you now, may cause too much effort. Maybe after you analyse your code and think about it, the effort might not be so high as it seems at a first glance. Maybe you can isolate most of the problems you have now and get your problems under control by using what you have now. But do not expect the silver bullet - framework and library decisions always have a certain risk of showing up to be wrong in the future, you are not the first one who made that experience.