Should automated unit tests be part of the build?

Question

Should automated unit tests be a part of the build process, or should they be manually run when someone makes changes to the code instead?

To me it seems like making it part of the build rather than leaving it a manual process has some clear advantages. Running your tests with the build would make them run more frequently and would keep them from getting of sync with the code.

The downside to running them with the build is that it would increase build times and could cause a build to fail, but I was under the impression that tests that take more than a couple hundredths of a second to complete are not good tests. Also if a build fails because of a test wouldn't that be a good thing? It would keep you from deploying broken code.

I don't see many people making the tests run directly as part of the build so I'm wondering why they are usually separate.

Answer 1

Disclaimer: I'm not a purist regarding testing. I like to do my TDD, but I don't think 100% coverage is necessary for all projects.

That being said, the rule of thumb should be "every change must be tested and approved before going to production".

That means you may run your tests after commit (triggered automatically by your continuous integration tool, like Jenkins, TFS or others) or during the build pre-deploy phase.

Personally, I find running tests for every single build I do on my machine boring and easy to make me "drift-off" to other chores while the tests are running.

That being said, no code that goes to production comes directly from my machine, they all go from a CI/Build machine. This guarantees that none of my code has the "Works only on my machine" stamp.

If you have a large team, testing after commit may be a better strategy, to allow you identify early the build-breakers and allow the team recover and communicate faster, but it'll probably need more server resources to do so.

With a smaller and well-oiled team testing pre-deploy to UAT / Production may be enough to guarantee quality.

Just keep in mind some rules of thumb:

• Uncommited code is non-existing code;
• Untested code should not go to production;
• Your machine should not produce the builds, a CI integration server should.

Answer 2

Should automated unit tests be a part of the build process, or should they be manually run when someone makes changes to the code instead?

Both.

UnitTests are a safety net for both, the individual developer and the integration process.

The individual developer verifies by running the UnitTests that the code he wrote really implements the desired behavior and does not break already existing desired behavior.

The integration process verifies by running the UnitTests that merging different developement paths did not break any desired behavior.

Answer 3

Make them a different build target: make unit. This way, it is up to the developer to decide whether the developer wants only compilation or also unit tests. Be sure to have mechanism of executing unit tests only for a given submodule, because it may not make sense to run all unit tests all the time.

As part of your continuous integration build (Jenkins or similar), run also all unit tests. And run them in a mode where one failure doesn't stop the rest of the tests. However, make unit can stop on the first failure.

Answer 4

I say, if your unit tests take less than 1-2 seconds to run, do it. Otherwise don't. Building your application should never take long IMO, because you'll lose focus every time you have to build.

Your build should fail if your unit tests don't pass. That'll enforce quality. What good is a build where some functionality doesn't work as intended?

Plus, it prevents you from forgetting to run your unit tests. Or it prevents you from voluntarily skipping the tests on a friday 5o'clock build.

Answer 5

Yes. :D But there is no one-size-fits-all answer.

But, proper test automation always detects more issues earlier than ad-hoc unit-testing, saving you from larger issues further on. And, remember - if you're not building for release, you're building to test your latest changes, aren't you?

Depending on the size of your project, you should choose to run all unit-tests as part of the private build, if the added time is neglectable. But adding more than 10s to a 1 minute-build, or 2s to a 1 second-build is probably not a good idea. In that case, you might want to identify one or two quick unit-tests that makes a sanity-test of the most important functionality that you run on each private build.

However, as soon as you have a build-server that is able to pull from the VCS, you should also add the functionality to run "all" unit-tests whenever a new successful build is produced. Unless "all" takes too long time, that is... It boils down to the number of builds/day, average build-time and unit-test time. As the project grows, you will find that you will need to make the build modular enough that you don't need to unit-test unrelated parts of the code-base.

And no matter how clever you design your test-cases, you will inevitably find an important use-case that really needs to take a long time - be it processing intensive or simply needing real time to tick. So, for those cases you will also need conditional, or manual, invocation.