How do we make unit tests run fast?

Question

We have reached the point in our project where we have almost a thousand tests and people have stopped bothering with running them before doing a check in because it takes so long. At best they run the tests that are relevant to the piece of code that they changed and at worst they simply check it in without testing.

I believe this problem is due to the fact that the solution has grown to 120 projects (we usually do much smaller projects and this is only the second time we do TDD properly) and the build + test time has grown to about two-three minutes on the lesser machines.

How do we lower the run time of the tests? Are there techniques? Faking more? Faking less? Maybe the bigger integration tests shouldn't run automatically when running all the tests?

Edit: as a response to several of the answers, we already use CI and a build server, this is how i know the tests fail. The problem (actually a symptom) is we keep getting messages about failed builds. Running partial tests is something that most people do but not all. and regarding the tests, they are actually pretty well made, they use fakes for everything and there is no IO at all.

Answer 1

A possible solution would be to move the testing portion from the development machines to a continuous integration setup (Jenkins for example) using version control software of some flavor (git, svn, etc...).

When new code has to be written the given developer will create a branch for whatever they are doing in the repository. All work will be done in this branch and they can commit their changes to the branch at any time without messing up the main line of code.

When the given feature, bug fix, or whatever else they are working on has been completed that branch can be merged back into the trunk (or however you prefer to do it) where all unit tests are run. If a test fails the merge is rejected and the developer is notified so they can fix the errors.

You can also have your CI server run the unit tests on each feature branch as commits are made. This way the developer can make some changes, commit the code, and let the server run the tests in the background while they continue to work on additional changes or other projects.

A great guide to one way of doing such a setup can be found here (git specific but should work for other version control systems): http://nvie.com/posts/a-successful-git-branching-model/

Answer 2

The majority of Unit Tests should take under 10 milliseconds each or so. Having 'almost a thousand tests' is nothing and should take maybe a few seconds to run.

If they're not, then you should stop writing highly coupled integration tests (unless that's what the code needs) and start writing good unit tests (starting with well decoupled code and proper usage of fakes/mocks/stubs/etc). That coupling will impact test quality and the time it takes to write them too - so it's not just a matter of reducing test run time.

Answer 3

There are several approaches I have used to solve similar issue:

1. Check execution time, and find all most slowest tests and then analyze why they take so much time to execute.
2. You have 100 projects, may be you do not need to build and test them every time? Could you run all unittest only at a night builds? Create several 'fast' build configurations for daily use. CI server will perform only limited set of unittests projects related to 'hot' parts of your current development process.
3. Mock and isolate everything you could, avoid disk/network I/O whenever it's possible
4. When it isn't possible to isolate such operations, may be you have integration tests? May be you could schedule integration tests to night builds only?
5. Check all occasional singletons, that keep references to instances/resources and that consume memory, this could leads to performance degradation while running all tests.

In addition you could use following tools to make both your life easy and tests run faster

1. Gated commit some CI servers could be configured to perform build and test before commiting code to source repository. If someone commits code without running all tests beforehand, that contains also failed tests, it will be rejected and returned back to author.
2. Configure CI server to execute tests in parallel: using several machines or processes. Examples are pnunit and CI configuration with several nodes.
3. Continuous testing plug-in for developers, that will automaticaly run all test during writing code.

Answer 4

0. Listen to your programmers.

If they're not running the tests, it means they perceive the cost (waiting for the tests to run, dealing with false failures) to be greater than the value (catching bugs right away). Decrease the costs, increase the value, and people will run the tests all the time.

1. Make your tests 100% reliable.

If you ever have tests that fail with false negatives, deal with that right away. Fix them, change them, eliminate them, whatever it takes to guarantee 100% reliability. (It's OK to have a set of unreliable, but still useful tests that you can run separately, but the main body of tests must be reliable.)

2. Change your systems to guarantee that all tests pass all the time.

Use continuous integration systems to ensure that only passing commits get merged in to the main/official/release/whatever branch.

3. Change your culture to value 100% passing tests.

Teach the lesson that a task isn't "done" until 100% of tests pass and it has been merged in to the main/official/release/whatever branch.

4. Make the tests fast.

I have worked on projects where tests take a second, and on projects where they take all day. There is a strong correlation between the time it takes to run tests and my productivity.

The longer tests take to run, the less often you'll run them. That means you'll go longer without getting feedback on the changes you're making. It also means you'll go longer between commits. Committing more often means smaller steps that are easier to merge; commit history is easier to follow; finding a bug in the history is easier; rolling back is easier, too.

Imagine tests that run so fast that you don't mind automatically running them every time you compile.

Making tests fast can be hard (that's what the OP asked, right!). Decoupling is key. Mocks/fakes are OK, but I think you can do better by refactoring to make mocks/fakes unnecessary. See Arlo Belshee's blog, starting with http://arlobelshee.com/post/the-no-mocks-book.

5. Make tests useful.

If the tests don't fail when you screw up, then what's the point? Teach yourselves to write tests that will catch the bugs you're likely to create. This is a skill unto itself, and will take lots of attention.

Answer 5

A couple minutes is OK for unit tests. However, keep in mind that there are 3 major types of tests:

1. Unit tests -- test each "unit" (class or method) independently of the rest of the project
2. Integration tests -- test the project as a whole, usually by making calls into the program. Some projects I've seen combine this with regression tests. There is significantly less mocking here than unit tests
3. Regression tests -- test the completed project as a whole, as the test suite is an end user. If you have a console application, you would use the console to run and test the program. You never expose internals to these tests and any end user of your program should (in theory) be able to run your regression test suite(even though they never will)

These are listed in order of speed. Unit tests should be quick. They won't catch every bug, but they establish that the program is decently sane. Unit tests should run in 3 minutes or less or decent hardware. You say you only have 1000 unit tests, and they take 2-3 minutes? Well, that's probably OK.

Things to check:

• Make sure to ensure that your unit tests and integration tests are separate though. Integration tests will always be slower.

• Ensure that your unit tests are running in parallel. There is no reason for them not to if they are true unit tests

• Ensure your unit tests are "dependency free". They should never access a database or the filesystem

Other than that, your tests don't sound too bad right now. However, for reference, one of my friend's on a Microsoft team has 4,000 unit tests that run in under 2 minutes on decent hardware(and it's a complicated project). It's possible to have fast unit tests. Eliminating dependencies(and mock only as much as needed) is the main thing to get speed.

Answer 6

Train your developers on Personal Software Process (PSP) helping them to understand and improve their performance by using more discipline. Writing code has nothing to do with slamming your fingers on a keyboard and afterwards press a compile and check in button.

PSP used to be very popular in the past when compiling code was a process which took a lot of time (hours/days on a mainframe so everybody had to share the compiler). But when personal workstations became more powerful, we all came to accept the process:

1. type some code without thinking
2. hit build/compile
3. fix your syntax to make it compile
4. run tests to see if what you wrote actually makes sense

If you think before you type, and then after you typed, review what you wrote, you can reduce the number of errors before you run a build and test suite. Learn not to press build 50 times a day, but maybe once or twice, then it matters less that your build and testing time takes a few minutes more.

Answer 7

One possible way: split your solution. If a solution has 100 projects, then it's quite unmanageable. Just because two projects (say A and B) use some common code from another project (say Lib) doesn't mean they have to be in the same solution.

Instead, you can create solution A with projects A and Lib and also solution B with projects B and Lib.

Answer 8

I am in a similar situation. I have unit tests which test communication with the server. They are testing the behavior with timeouts, cancelling connections etc. The whole set of tests runs 7 minutes.

7 minutes is a relatively short time but it's not something you will do before every commit.

We also have a set of automated UI tests, their run time is 2 hours. It's not something you want to run every day on your computer.

So, what to do?

1. Changing the tests is usually not very effective.
2. Run only the relevant tests before your commit.
3. Run all of your tests every day (or several times a day) on a build server. This will also give you the possibility to generate nice code coverage & code analysis reports.

The important thing is: all of your tests should be run often because it's important to find the bugs. However, it's not absolutely necessary to find them before the commits.

Answer 9

Though your description of the problem does not give a thorough insight into the codebase, I think I can safely say your problem is two-fold.

Learn to write the right tests.

You say you have almost a thousand tests, and you have 120 projects. Assuming that at most half of those projects are test projects, you have 1000 tests to 60 production code projects. That gives you about 16-17 tests pr. project!!!

That is probably the amount of tests that I would have to cover about 1-2 classes in a production system. So unless you only have 1-2 classes in each project (in which case your project structure is too fine grained) your tests are too big, they cover too much ground. You say this is the first project that you are doing TDD properly. A say, the numbers that you present indicate that this is not the case, you are not doing TDD property.

You need to learn to write the right tests, which probably mean that you need to learn how to make the code testable in the first place. If you cannot find the experience inside the team to do that, I would suggest hiring help from the outside, e.g. in form of one or two consultants helping your team over a duration of 2-3 months to learn to write testable code, and small minimal unit tests.

As a comparison, on the .NET project that I am currently working on, we can run roughly about 500 unit tests in less than 10 seconds (and that was not even measured on a high spec machine). If those were your figures, you would not be afraid to run these locally every so often.

Learn to manage the project structure.

You have divided the solution into 120 projects. That is by my standards a staggering amount of projects.

So if it makes sense to actually have that amount of projects (which I have a feeling it doesn't - but your question does not provide enough information to make a qualified judgement of this), you need to divide the projects into smaller components that can be build, versioned, and deployed separately. So when a developer runs unit the test suite, he/she only needs to run the tests relating to component he/she is working on currently. The build server should take care of verifying that everything integrates correctly.

But splitting up a project in multiple components build, versioned, and deployed separately requires in my experience a very mature development team, a team that is more mature than I get the feeling that your team is.

But at any rate, you need to do something about the project structure. Either split the projects into separate components, or start merging projects.

Ask yourself if you really need 120 projects?

p.s. You might want to check out NCrunch. It's a Visual Studio plug-in that runs your test automatically in the background.

Answer 10

JUnit test are normally to be quick, but some of them simply must take some time to excecute.

For example, the database test are usually taking a few time to initialize and finish.

If you have hundreds of tests, even if they are fast, they require a lot of time to run because of their number.

What can be done is:

1) Identify the crucial tests. Those for the most important parts of libraries and those which are most likely to fail after changes. Only those test should be run always on compile. If some code is often broken, its tests should be obligatory even if they take long to execute, on the other side, if some part of software never caused a problem, you can safely skip tests on each build.

2) Prepare the continuous integration server, which will run all tests in the background. It's up to you if you decide to build every hour or to build after every commit (the second makes sense only if you want to automatically detect whose commit has caused trouble).

Answer 11

Issues I have seen:

a) Using IOC to build up test elements. 70 seconds -> 7 seconds by removing Container.

b) Not mocking out all classes. Keep your unit tests to a single element. I have seen tests that ramble through a couple of classes. These are not unit tests and much more likely to break.

c) Profile them to find out what was happening. I found the constructor was building stuff I did not need so I localised it and reduced run times.

d) Profile. perhaps the code is not that good and you can gain some efficiency out of a review.

e) Remove dependencies. Keeping you test executable small will reduce the time to load. Use an interface library and IOC containers to run your final solution but your main test projects should only have the interface library defined. This ensures separation, ensures it is easier to test, and also makes your test foot print smaller.

Answer 12

I feel your pain, and I have run into several places where the build speed can be improved a lot. However, the number on thing I recommend is to measure at a granular detail to figure out where your build is taking the longest. For example, I have a build with around 30 projects that takes just over a minute to run. However, that is only part of the picture. I also know which projects take the longest to build, which helps focus my efforts.

Things that eat up build time:

• Package downloading (Nuget for C#, Maven for Java, Gem for Ruby, etc.)
• Copying large amounts of files on the file system (example: GDAL support files)
• Opening connections to the database (some take over a second per connection to negotiate)
• Reflection based code
• Autogenerated code
• Using exceptions to control program flow

Mock libraries either use reflection or inject code using bytecode libraries to generate the mock for you. While it's very convenient, it eats up test time. If you are generating mocks inside a loop in your test, it can add a measurable amount of time to the unit tests.

There are ways to fix the problems:

• Move tests involving a database to integration (i.e. only on the CI build server)
• Avoid creating mocks in loops in your tests. In fact just avoid loops in your tests altogether. You can probably get the same results using a parameterized test in that case.
• Consider splitting your massive solution into separate solutions

When your solution contains over 100 projects, you have a combination of library code, tests, and application code. Each of the libraries can be it's own solution with it's associated tests. Jet Brains Team City is a CI build server that doubles as a Nuget server--and I'm sure it's not the only one. That gives you the flexibility to move those libraries that probably don't get changed often to their own solutions/projects and use Nuget to resolve the dependencies for your application code. Smaller solutions mean that you can make your changes to a library quickly and without any fuss, and enjoy the benefits in the main solution.

Answer 13

Can your test environment run anywhere? If it can, use cloud computing to run the tests. Split the tests among N virtual machines. If the time to run the tests on a single machine is T1 seconds, then the time to run them split up, T2, could approach T2=T1/N. (Assuming each test case takes about the same amount of time.) And you only have to pay for the VMs when you're using them. So you don't have a bunch of test machines sitting in some lab somewhere 24/7. (I'd love to be able to do this where I work, but we're tied to specific hardware. No VMs for me.)