What can you do about the quality of existing integration and unit tests while being the new guy in a team ?

Question

A recurring theme that I came across in my career is being the new developer to arrive in a team, and quickly having an inherent distrust of the existing unit and integration test suites.

During the interview you are told by the management that they "strongly support unit testing" and that they openly encourage it. They do, but everything about the tests themselves is just plain wrong. Like the fact that they are claiming 100% coverage when there is 100% integration test coverage but less than 10% repeatable unit test coverage. Some other problems I have found:

1. No clear indication between what is a unit test and what is an integration test. Unit and integration tests are mixed together in the same class.

2. Integration tests that have undeclared explicit dependencies on very specific dynamic data on a specific environment's database.

3. Non-transactional integration tests, basically tests that may or may not bother to clean up after themselves, sometimes requiring manual database "scrubbing" to make the test repeatable.

4. No mocking whatsoever, and application code requires a major overhaul just for mocking to be possible. In other words, design without testing in mind.

5. No clear naming conventions to quickly look at a test name and determine roughly what tests are being done.

This is all not to say that ALL of the tests are useless or bad, a good deal of them are pretty good and worth keeping, but it feels like panning for gold sometimes. I would purposely avoid running tests just because I was afraid of screwing up the database for my black-box test cases.

This has essentially given me an inherent mistrust of unit and integration tests that I have not personally written or reviewed in some way. At some level if you don't have faith in the quality of your test suite then it really brings no value to the team or the project whatsoever.

What do you do when you find yourself in this situation? What do you feel the best plan of attack would be to tackle something like this?

Should all of the tests be refactored in a monumental effort spanning across releases? Should you just abandon the idea that this legacy project may have one day solid unit test coverage?

Answer 1

First of all, as other posters have already wrote, you should be happy unit/integration testing is understood (not in a technical way, I mean the reason why it should be done is understood) and pushed by the management.

Before you start doing everything you should expose the problems to the management, why something should be done, and very diplomatically so that they don't think you think you are the best programmer in the world (even if you are! :-). Maybe they will tell you that the application is going to be replaced by something completely new, and if so, why bother. And maybe they will realize that it would be nice and speed up the testing phase before each release. And be diplomatic with your teammates, there may be million reasons why it is as it is and there is just no reason to look for a culprit.

A better idea would be to try to talk to them so that they can participate in the effort, and you will have fewer chances to appear as a smart-ass, it would be more "we" than "I".

Now, put priorities on what you want to do. Prioritize the tasks, always knowing that your first priority will always be your current project assignment... As for your test problem, here is what I would do, in three phases:

1. How to make a difference between unit and integration tests? The following solutions may apply and are not exclusive:

   • Refactor the name of the test case methods (not the classes since the right behavior is to have the test case share the same name as the tested class)
   • Create two annotations, one named "UnitTest", the other "IntegrationTest". These annotations can be used on classes and on methods. If an entire class is composed of either unit or integration tests, you can mark the class with the right annotation. If not, you can mark each method with the right annotation. Also, these annotations could be useful to dynamically inject fixtures before launching the tests (phase 3)

2. For each integration test, list what is the "dataset" that is expected to be in the database at the beginning of each test, and what should be removed at the end of it (ex: table X, needs a record with "id" set to "1", and "name" set to "foo", etc.). Note that what you remove may be bigger/smaller than what you have at the beginning since the code itself may respectively add/remove objects from the persistence layer. You will most probably quickly notice that several of these test cases need the same dataset or part of the same dataset.

3. The first two phases can be done relatively quickly... compared to the rest. Now that you have your dataset, you use create dataset fixtures for each test case that needs it. But that will cost some time... So you could do one, see how much time it cost you and estimate how much more time you need to do everything. Also, you could use that test to demonstrate to your fellow colleagues what you did, and why life is so much easier when you don't need to have a database in a specific state juts to perform tests.

You may note that phase 1, 2 and 3 may be done on a single test class, if you quickly want to show colleagues/management how it can be done. This would also be useful, as Péter Török wrote, in order to immediately show the "good way" to your teammates so that they stop producing bad code. However I think that rest of the phase 2, identifying the entire test dataset, is best done in one big step.

As for the API/technology behind all that, you seem to know the subject.

Hope that helped a little bit.

Note: for my proposal, I am supposing you code either in Java/C# where I know annotations and AOP is possible. I'm sure this is also possible in other languages, but I won't write about something I don't know.

Answer 2

You will not be able to fix all the tests together. I think you should focus on the word improvement versus overhaul. Neither management not developers will agree on an overhaul but if you show that there is a way to improve things without affecting the project negatively, they will be more likely to listen.

First, you cannot 'fix' or refactor existing code unless you have quality test coverage, so I would focus on fixing your testing infrastructure first.

Make a list of things that need improvement and attempt to prioritize them. I think the ability to run tests independently and singly (so they do not effect each other) and the separation of unit and integration tests are some of the first things to work on. You need to make it easy for you and others to do the right thing.

As far as application code is concerned... You won't be able to do the complete overhaul of the application architecture just so it can be better unit tested. Instead, when you put in new code, try to apply principles that make it easier to unit test (like dependency injection). You might think that this is not big enough of a change, but it is amazing how fast developers catch on if they see the benefit. There just needs to be someone who starts making a change for the better.

Talk to your team and get them to buy-in. One of the most important things you can do is follow the 'Boy Scout Rule' and make small improvements to leave a class or test in a better shape then you found it. If the whole team applies this rule, things get better a lot faster.

After a while, you will have a lot of code that follows good principles and areas of the application that are in bad shape. At this point, you have a baseline of what good is and the team can decide if its beneficial to do a larger refactor for the older legacy stuff or it can just live on as it is.

Answer 3

I must admit I would take it as a rare gift to arrive at a project where they already have a significant number of unit/integration tests - I have never had that luck in my life so far. Even if not all of them are working as they should, there is already a great deal of effort put into it, and even if the team and management don't always follow best practices, still they already are committed to the cause, so you don't need to spend your time arguing about why unit tests are worth writing.

However, the tests as you describe them can indeed take some improvement. But you need to be careful with your tone when discussing the problems with your teammates. If you start by stating "everything about the tests themselves is just plain wrong", you may very quickly end up alienating the rest of the team. Instead, you should focus on how to improve the current state of affairs, which - I must repeat - is still significantly better than the average according to my experience so far.

IMO your first target should be to stop the team from producing more bad tests. So start by demonstrating the benefit of each specific improvement to your teammates. If they see the usefulness of a certain technique - be it cutting external dependencies, restoring the original state after each test, or separating unit and integration tests - they will start applying them. This in itself will slowly but surely improve the overall quality of the test base in the long run (if you have 1000 bad test cases now, and the team produces 1000 good tests by next year, you will have brought down the ratio of bad tests from 100% to 50%). Once that is secured, you can decide about refactoring existing tests on a case by case basis. Again, small improvements will add up to big changes over time.

As a side note, from the tone of your post I feel you might be in a place where I used to be too: not trusting in work done by others, unable to delegate tasks to anyone in fear of their work not being up to your own quality standards. In my experience, this is not a good place to be, because in the long run, it may easily lead to personal conflicts and burnout. You can't do everything alone, you must work together with the rest of the team. You can only succeed together.

Answer 4

Work towards test independence. If test X modifies the test database in such a way that test Y fails, then change test Y. In this little scenario the thing to focus on is not that X has messed up the database, but rather that Y has inappropriate dependencies. Remove those dependencies (by stubbing out the database, by restructuring the test, or by initializing the database to a state where Y will pass) and now you have one more independent, working test. That's progress (arguably, "fixing" X not to mess up the database would not be).

Be patient and respectful, as best you can, of the folks you're working with, despite the mess they've created. They're trying to do the right thing, in all likelihood; keep that in mind and help them to do it.

Answer 5

The good thing about being new to the team is that you have a "fresh" approach to things. The bad part may be that others may have a hard time believing you.

Don't make a laundry list of things to be done. But pick ONE thing that seems urgent, and that others are most likely to respond to, and suggest a solution. If it works, great, then suggest another solution to another problem, on the strength of your first success.

But take it slow, one by one, and hope that your ideas slowly "catch on" among the new group.

Answer 6

set the example you want to see, but appreciate another developer's perspective in a test: one developer may test for successes, while another may test for failures, one developer wrote the class while another may have used it the first time when writing the test.

the existing tests (erm, most) still have purpose, although it may not be immediately evident. i don't recommend overhauling and refactoring everything all at once (it is tedious and error prone). bad tests ultimately need to be updated, rewritten, or they may simply become obsolete as software evolves.

if your approach to testing is superior in some regards, people will learn from that model or ask you for your assistance. there will be a balance of resources, but you can extend as needed to support your tests.

ultimately, you want to increase the quality of the programs via tests and improve coverage. you can do this by placing more emphasis on testing and setting a good example - but do appreciate other's perspectives.

when emphasizing the importance, you can make your tests' operational environment different - one obvious example is to run tests in parallel; if it's not reentrant, you have located a bug in your program or test (win/win). a more rigorous testing environment will force the bad tests to be fixed and/or rewritten (hopefully the right way the second time around). slowly introduce such changes (don't break half the tests all at once -- that's a real problem), forcing them to learn what makes a good test, and ultimately a good program. then when you and other go in and modify/repair/extend tests, apply what you have learned -- it's incremental and you have a better understanding of the context/program at the time.

Answer 7

Fix them over time

I've been in the same situation. I just spent an hour every morning going through tests and fixing them until they were all fixed. It was a medium sized codebase and I think I finished in 1.5 months. I also got much more confident in our tests.

I personally don't care if a test is a unit test or integration test as long as it's a good test. By good test I mean that it:

• cleans up after itself
• is repeatable
• minimizes environmental interactions
• is consistent

Along the way I evangelized better test construction (which is to say I bugged people alot).