22

I'm working in the development of an old project written in Java. We have more than 10 million LOC and, even worse, more than 4000 functional tests.

The tests, scheduled by Hudson, are failing like mad with every bigger code change. Verification of the test failure - if it's a problem in the product or in the test, takes months. We cannot remove the old tests because we don't know what they are testing!

What we can do? How to proceed with such amount of legacy tests?

closed as too broad by gnat, durron597, user22815, user40980, Robert Harvey Sep 17 '15 at 21:34

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 6
    Real questions have answers. Rather than explaining why your situation is terrible, or why your boss/coworker makes you unhappy, explain what you want to do to make it better. For more information, click here... – gnat Sep 17 '15 at 8:35
  • 13
    Why did you allow tests to start failing in the first place? BTW 4000 is not that many tests for 10 MLOC – BЈовић Sep 17 '15 at 8:39
  • 6
    Stop, drop, and roll. – Navin Sep 17 '15 at 9:44
  • 13
    Find out what the tests are testing. Then revisit and wonder first of all how on earth tests take months to find a problem in, and also find out how your requirements changed so much. Tests are meant to encapsulate requirements in an application. If your tests fail, your code is not performing according to requirements - either you wrote them incorrectly or none of your code adheres to your requirements. – Dan Pantry Sep 17 '15 at 10:10
  • 6
    We've all seen a compiler kick out a zillion errors because of a single missing '}'. If these are functional tests with a plethora of dependencies, perhaps the same sort of problem is at work? – Dan Pichelman Sep 17 '15 at 13:50
37

Abandon them.

I know it's hard to let go of something that was clearly a lot of effort to produce, but the tests aren't working for you, they're working against you. A test suite is supposed to give you confidence that the system does what it's supposed to do. If it doesn't do that, it's a liability instead of an asset. It doesn't matter whether the system or the tests are at fault - as long as running the test suite signals huge amounts of errors, it cannot fulfill its purpose.

What you need now is a new suite of tests that runs with no errors. That means it will initially have little coverage, in fact almost no coverage. Every time you fix or take the time to thoroughly understand something about your system, you reord that knowledge in a test. Over time, this will produce a new safety net that you can build on in the future. Trying to patch an old, ill-understood safety net is a time sink that is almost never worthwhile.

I would even advocate against transferring tests from the old suite to the new suite. Sure, some of them may succeed now, but is that because they are testing exactly what they are supposed to test, or just because some random shots always hit the target? Obviously you have to be pragmatic about what can and can't be done with the effort you have avaiable to spend, but you can't compromise on the principle that a test suite must run cleanly in order to do its job.

  • 9
    I can't see the logic in your point: "A test suite is supposed to give you confidence that the system does what it's supposed to do. [...] What you need now is a new suite of tests that runs with no errors." If you have faulty code that makes tests fail doesn't mean you should re-write the tests so that the faulty code passes. – DBedrenko Sep 17 '15 at 13:01
  • 13
    Hector's situation is that he doesn't know whether the code or the tests are wrong. If he did, he could work with the code base and change sometime the tests, sometimes the business code. As it is, even this kind of drudgework wouldn't pay, since you don't know whether you're fixing problems or perpetating them. – Kilian Foth Sep 17 '15 at 13:06
  • 5
    "A test suite is supposed to give you confidence that the system does what [it should]." No, it's supposed to tell me whether the system does what it should; false confidence is worse than none. "What you need is a test suite that runs with no errors" No, what he needs is a test suite that gives him useful information about the soundness of the code. What he has now is many cryptic warning lights, which are better than a green light from a shiny new test suite that doesn't test anything. He should disable the old tests temporarily, but not abandon any that he hasn't verified as spurious. – Beta Sep 17 '15 at 16:45
  • 4
    This answer is incredibly bad advice! If smaller code changes leads to a large amount of failed tests you probably have code quality issues. Test will at least notify you that you broke something. You need to improve the code (by carefully refactoring aided by tests). If you just remove the tests you have no way of knowing if you break something. – JacquesB Sep 17 '15 at 18:06
  • 4
    This is terrible advice. If the OP and his team already can't understand the codebase and it's tests, throwing out the tests and starting over is unlikely to solve the OP's core problem - understanding the codebase. I think we can assume the tests worked when written - so his team needs to track down what each test is testing, and read the source to determine if it's the codebase or the test which is wrong today. Far simpler than starting over with mis-guided and un-informed/naive tests. – SnakeDoc Sep 17 '15 at 18:38
29

Go and fix the tests.

Your biggest mistake is that you allowed tests to fail, and you obviously ignored it for a while. What you have is not "legacy tests" - you are working on a legacy code. And I consider every code written without tests to be legacy.


Verification of the test failure - if it's problem in product or in test, takes months. We cannot remove the old tests because we didn't know what they are testing!

It looks like there is a even bigger problem in your organization, since you are not working with clear requirements. I can not understand that you (or someone else) can't confirm the right behaviour.

  • 4
    That's what should ideally be done, but it seems that the tests here are so bad that programmers don't even know what they are testing. I think that in this case it may be best to get rid of the WTF tests and start writing new and meaningful ones right away! In a recent project I had a similar problem with a coworker whose tests always failed for no good reasons (it didn't fail because what was supposed to be tested got wrong, but because the test code was so brittle and not even deterministic!). I spent days rewriting what I could, and trashed the rest! – Shautieh Sep 17 '15 at 10:22
  • @Shautieh The WTF tests do not go without WTF code, so fixing tests usually means refactoring the code. And randomly failing tests are the sign of incompetence. And the supervisor of your coleague is to blame for not doing their job. – BЈовић Sep 17 '15 at 11:37
  • 2
    Sometimes life is harsh: the guy responsible for the WTF tests (and code) gained the highest salary in the team (20+% more than me), and when he quit in the middle of the project (because he found a higher paying job) I had to take upon some of his devs :/ But you are absolutely right to say our supervisor was to blame too ^^ – Shautieh Sep 17 '15 at 16:04
  • @Shautieh: a colleague of mine once said that a bug in the code is two bugs: a bug in the code and a blind spot in the tests. I guess it's actually three if you count the developer who tolerates failing tests, and four if you count the managers who promote such an incompetent. – Beta Sep 17 '15 at 16:48
  • @Beta Sounds quite similar to the definition sometimes used in TDD: "A bug is a test which you haven't written yet." – Angew Sep 17 '15 at 17:00
22

Tests are valuable. At the very least, they record that somebody considered that they should spend time writing them, so presumably they had some value to somebody once. With luck, they will contain a complete record of all the features and bugs that the team has ever worked on – though they may also just have been a way to hit some arbitrary test coverage number without being carefully thought through. Until you look at them, you won't know which is the case here.

If most of your tests pass most of the time, then just bite the bullet and invest the time in figuring out what the few tests that fail were trying to do, and either fixing or improving them so that the job will be easier next time. In that case, skip forward to the Determine the intent for each test section, for some advice on what to do with a small number of failing tests.

On the other hand, you may be faced with a Red build now, and hundreds or even thousands of tests which haven't passed for a while, and Jenkins hasn't been Green for a long time. At this point, the Jenkins build status has become useless, and a key indicator of problems with your check-in is no longer working. You need to fix this, but can't afford to stop all forward progress while you tidy up the mess in your living room.

In order to keep your sanity while performing the required archaeology to determine what value can be recovered from the tests which have been failing, I'd recommend the following steps:

Temporarily disable the failing tests.

There are several ways you could do this, depending on your environment, which you don't clearly describe so I can't really recommend any particular one.

Some frameworks support the notion of expected failures. If yours does, then this is great, as you'll see a countdown of how many tests are left in this category, and you'll even be informed if some of them start passing unexpectedly.

Some frameworks support test groups, and allow you to tell Hudson only to run some of the tests, or to skip a group of the tests. This means you can occasionally run the test group manually to see if any are now passing.

Some frameworks allow you to annotate or otherwise mark single tests to be Ignored. It is harder to run them as a group in this case, but it stops them from distracting you.

You might move the tests to a source tree that isn't normally included in the build.

In extremis, you could delete the code from the HEAD of the version control system, but this will then make it harder to recognise when the third phase has been completed.

The goal is to get Jenkins to go Green as soon as possible, so you can start moving in the right direction as soon as possible.

Keep tests relevant.

Resolve to add new tests as you add or modify code, and commit to keeping all the passing tests passing.

Tests may fail for a variety of reasons, including the fact that they weren't well-written tests to start with. But once you get Jenkins green, keeping it that way is really important.

Get used to writing good tests, and make it a big deal if tests start failing.

Determine the intent for each test.

Go through the disabled tests one by one. Start with the ones that affect the modules that you change most frequently. Determine the intent of the test, and the reason for failure.

  • Does it test a feature that was removed from the code base on purpose? Then you can probably delete it.

  • Is it catching a bug that nobody has noticed yet? Reinstate the test and fix the bug.

  • Is it failing because it was making unwarranted assumptions (e.g. assuming button text would always be in English, but now you have localised your application for multiple languages)? Then figure out how to make the test focus on a single thing, and isolate it from unrelated changes as best you can.

  • Does the test sprawl through the whole application, and represent a System test? Then remove it from your main Jenkins test suite and add it to the Regression suite that runs less frequently.

  • Has the architecture of the app changed beyond recognition, so the test no longer does anything useful? Delete it.

  • Was the test added to artificially increase code coverage statistics, but actually does nothing more than confirm that the code compiles correctly and doesn't go into an infinite loop? Or else, the test simply confirms that your selected mocking framework returns the results you just told it to? Delete it.

As a result of this, some tests will stand, some be modified, some split into multiple independent, bite-size chunks, and some removed. As long as you are still making progress with new requirements, setting aside a little time to deal with technical debt like this is the responsible thing to do.

  • 1
    It is a really, really bad idea to disable tests just because they fail! The rest of your advice are good, but not this. Tests you don't understand should never be disabled. The point of testing is not to get a green bar, the point is to get working software! – JacquesB Sep 18 '15 at 8:56
  • It depends on the scale of the problem. But I agree, I haven't actually made that clear. – Bill Michell Sep 18 '15 at 9:11
  • Added a paragraph to differentiate between "we're green but every change makes stuff go red" and "we've been red so long, we've forgotten what green looks like" – Bill Michell Sep 18 '15 at 9:20
  • Instead of disabling or even deleting the test, some frameworks also provide the notion of an expected failure. This could help increase the SNR because you'll be more directly alerted about new failures (which you won't if there is alsways a huge number of failures) but still be notified about the known failures and – perhaps even more important – when a previously failing test suddenly passes again. If the unexpected failures are read and the expected failures orange, then make making the red tests green your first and making the orange ones green your second priority. – 5gon12eder Sep 18 '15 at 9:55
11

4000 tests is an intractable problem. 40 tests is more tractable. Randomly select a manageable number of tests to run and analyze. Classify the results as:

  1. Useless test
  2. Useful test that runs clean
  3. Useful test that fails

If a lot of the tests fall in the first category, it may be time to throw out your current test suite and put together a useful one for the current code.

If a lot of the tests are failing in ways that tell you about a problem in your code, you need to work through the failing tests fixing things. You may find that fixing one or two bugs makes a large number of tests run.

  • 2
    +(int)(PI/3) for providing an actual & simple way of testing the test suite - while I agree that, as a rule of thumb, tests such as described by OP are a sign of a faulty design - but without testing what's wrong, any advice on the test suite itself (be it "abandon them", "fix the tests", "write new tests") is just plain useless. Exactly as you say: if I'd have 4k tests and for 40 completely random of those 3/4 are crappy and useless - I wouldn't hestitate to dump the whole suite. If 3/4 of those would have actually been useful - I'd leave'em & focus on improving the code. – vaxquis Sep 17 '15 at 18:15
7

If this statement is true,

The tests ... are failing like mad with every bigger code change.

then that implies that if you rollback to the code just before a "bigger code change", then many of the tests will pass again. After doing that, grab a smaller chunk of changes and see which tests are newly failing. This will help you better isolate which code changes are causing which tests to fail. For each test, once you have isolated the problem, you should then be able to determine if the new code was flawed, or the test was. If it is a problem with the new code, make sure to compare it with the latest version just in case that particular bug has already been fixed.

Repeat until you have the latest code base.

This may seem like an overwhelming task, but it's very likely that once you go down this path and begin isolating some of the problems, a pattern will begin to emerge which may greatly speed up the process. For example:

  • You may notice that many tests are dependent on something else that is flawed. Fixing that one piece may fix many tests.
  • You may notice that many tests are flawed and need to be fixed or removed.
  • You may notice that a particular developer has a much higher frequency of causing tests to break. That developer may need more training or supervision.
3

If you don't know what they are testing, remove them until you do know. Tests are fluid things, if you remove a feature that is no longer required, then you should expect to have to change the test that tests that feature! So unless you know what the tests are testing, you have no hope of changing the codebase with them in place.

You can get the test system set up on developer's machines and run there so the devs can see what parts the tests are interacting with, hopefully provide this missing documentation, and become more familiar with the codebase that you are either not changing correctly, or no longer testing correctly.

In short - if your old tests are failing when you make changes, your code changes are not good. Use those tests as a means of education in how the system works.

  • 1
    This is why I like JUnit's @Ignore annotation -- you can keep your tests, but not execute them. Then it's simply a matter of re-enabling them and fixing them one at a time. It allows you to narrow your focus down to just a handful of tests at a time, instead of being overwhelmed with thousands of failures. – TMN Sep 17 '15 at 12:55
  • 1
    This is bad advice. You should not remove or disable a test you don't understand. Only if you do understand the test, and you are confident it tests an obsolete feature, should it be disabled or removed. – JacquesB Sep 18 '15 at 8:53
2

The most important thing I would do is go back to the fundamentals of what testing is supposed to do, and what the business needs to keep moving. Testing's job is to identify issues before they become expensive to fix later. I think the key word in that sentence is "expensive." These issues need a business solution. Are expensive issues showing up in the field? If so, testing is failing outright.

Your management and you need to come to a reality check. You are finding that development costs are skyrocketing because of a legacy set of tests. How do those costs compare to the costs of delivering a faulty product because you disabled tests? How do they compare to the onerous task of actually figuring out what behaviors users need (which are the things that should be tested)?

These are issues that need business solutions because they touch the business side of the job. You are delivering product to a customer, and that is a boundary which business is very interested in. They may be able to identify solutions that you, as a developer, cannot. For example, it may be reasonable for them to provide two products: one "legacy" product for those who need reliability and are willing to forgo new features, with one "visionary" product which may have more faults, but is pioneering ahead. This would give you an opportunity to develop two independent sets of tests... one legacy one with 4000 tests, and one with more of the tests you think need to be done (and document them so this process doesn't repeat).

Then, the art begins: how can you manage this two-headed beast so that advances in one branch also help the other branch? How can your updates to the "visonary" branch flow back towards the "legacy" branch, despite rigid testing requirements. How can continued customer requests on the "legacy" branch better shape your understanding of the requirements your legacy customers would need if you eventually re-merged the products?

-3

We cannot remove the old tests because we didn't know what they are testing!

That's exactly why you should remove the old tests! If you don't know what they are doing, then failure is meaningless and running them is a waste of time. Throw them out and start over.

  • 2
    this seems to merely repeat point already made and explained in top answer – gnat Sep 17 '15 at 16:17
  • 4
    Failure is not "meaningless", it means you don't understand the system as well as you thought you did. – Ben Voigt Sep 17 '15 at 19:10
  • Failure is definitely meaningless here, because the OP clearly stated they don't understand the system. – Mohair Sep 17 '15 at 22:19

Not the answer you're looking for? Browse other questions tagged or ask your own question.