I have a set of about 300 unit tests that have been through a difficult few months. The poor tests were subject to being upgraded from a V110 compiler (Visual Studio 2008) alongside Visual Studio 2012 to a V120 (Visual Studio 2013) compiler alongside an upgrade to Visual Studio 2015.
These tests were passing in VS2012. They broke horribly after the compiler/IDE upgrade. Some things were very odd, I spent weeks trying to fix them, worked with several engineers, and found very little traction. Then a beam of hope came in the form of VS2015 Update 3. It fixed the tests. For about 2 days. Now I'm met with a dreaded, "Failed to set up the execution context to run the test", for a test that was working last week.
So, the tests have been a bit battered recently. These are critical tests that my team needs to be running (and that we can't put in the gated builds because of the failing tests). We have workarounds to run the tests piece-by-piece, in an older IDE, from the command line, or any other number of tricks and hacks...but we really can't have tricks and hacks for important unit tests.
I'm not asking how to fix this specific set of tests, but moreover how should I as a software engineer start to regain stability with a set of unit tests that became unstable. As I'm writing this the answer seems obvious: Snap off the smallest subset of failing tests and analyze the failures/start the debugging process (which of course I'm doing), but things always have a rub. The upgrade happened, and there's no going back, so at the moment we're stuck with some (occasionally) unreliable tests.
Because the tests can be (not easily and not always all of them) ran, fixing these tests doesn't ever seem to be a higher priority for my team than pushing out new features. I'm an SD1 and so I don't have a lot of sway to get the more senior members of the team to analyze these test failures and I feel like I'm spinning my wheels trying to fix tests while waiting for the next test/tool failure. What should I do?