1

Should automated unit tests be a part of the build process, or should they be manually run when someone makes changes to the code instead?

To me it seems like making it part of the build rather than leaving it a manual process has some clear advantages. Running your tests with the build would make them run more frequently and would keep them from getting of sync with the code.

The downside to running them with the build is that it would increase build times and could cause a build to fail, but I was under the impression that tests that take more than a couple hundredths of a second to complete are not good tests. Also if a build fails because of a test wouldn't that be a good thing? It would keep you from deploying broken code.

I don't see many people making the tests run directly as part of the build so I'm wondering why they are usually separate.

  • As you wrote yourself, there are benefits and downsides. So how can there be a general answer? – Frank Puffer Jul 20 '17 at 19:14
  • @FrankPuffer, I personally feel like the downsides are not really valid and was looking to see if someone had a better explanation of why it isn't done everywhere. – tjwrona1992 Jul 20 '17 at 19:17
  • A build can be useful even if some tests fail. Not every build is intended to be delivered to the customer. And a failing test does not always mean that the tested code is broken. I have seen a number of cases where the tests were simpliy too strict and sometimes simply didn't make sense at all. – Frank Puffer Jul 20 '17 at 19:32
  • In that case shouldn't the test be fixed before continuing? Tests are equally important as the code to be correct. If you just start ignoring failed tests you'll end up with a mess of failures everywhere. – tjwrona1992 Jul 20 '17 at 19:37
  • Possible duplicate of How do we make unit tests run fast? – gnat Jul 20 '17 at 19:41
5

Disclaimer: I'm not a purist regarding testing. I like to do my TDD, but I don't think 100% coverage is necessary for all projects.

That being said, the rule of thumb should be "every change must be tested and approved before going to production".

That means you may run your tests after commit (triggered automatically by your continuous integration tool, like Jenkins, TFS or others) or during the build pre-deploy phase.

Personally, I find running tests for every single build I do on my machine boring and easy to make me "drift-off" to other chores while the tests are running.

That being said, no code that goes to production comes directly from my machine, they all go from a CI/Build machine. This guarantees that none of my code has the "Works only on my machine" stamp.

If you have a large team, testing after commit may be a better strategy, to allow you identify early the build-breakers and allow the team recover and communicate faster, but it'll probably need more server resources to do so.

With a smaller and well-oiled team testing pre-deploy to UAT / Production may be enough to guarantee quality.

Just keep in mind some rules of thumb:

  • Uncommited code is non-existing code;
  • Untested code should not go to production;
  • Your machine should not produce the builds, a CI integration server should.
  • "I don't think 100% coverage is necessary for all projects." That's true for code coverage but when talking about UnitTests we should look for requirements coverage which indeed should be 100%. Unfortunately there is no technical possibility to measure this... – Timothy Truckle Jul 20 '17 at 19:26
  • 1
    @TimothyTruckle, on that I agree 100% with you. If you have a measurable or verifiable requirement, that definitely should be tested. – Machado Jul 20 '17 at 19:27
  • 1
    Thanks that helps, I think the main problem is we don't have a CI tool. I will have to look into using Jenkins to do our builds for us. – tjwrona1992 Jul 20 '17 at 19:52
  • @tjwrona1992, the initial setup of a CI tool usually is a pain in the butt, but it's worth a lot in the long run. You'll make a lot of mistakes on the first project, but take my advice: it'll pay itself in both cost, time and quality of life of your developers sooner than you expect. – Machado Jul 20 '17 at 20:59
2

Should automated unit tests be a part of the build process, or should they be manually run when someone makes changes to the code instead?

Both.

UnitTests are a safety net for both, the individual developer and the integration process.

The individual developer verifies by running the UnitTests that the code he wrote really implements the desired behavior and does not break already existing desired behavior.

The integration process verifies by running the UnitTests that merging different developement paths did not break any desired behavior.

  • I was about to enter this as an answer, but you beat me too it. I agree with this 100%. You should be testing at both stages. – Rhys Johns Jul 20 '17 at 23:47
0

Make them a different build target: make unit. This way, it is up to the developer to decide whether the developer wants only compilation or also unit tests. Be sure to have mechanism of executing unit tests only for a given submodule, because it may not make sense to run all unit tests all the time.

As part of your continuous integration build (Jenkins or similar), run also all unit tests. And run them in a mode where one failure doesn't stop the rest of the tests. However, make unit can stop on the first failure.

  • This makes sense, however we don't currently use Jenkins. We currently just use a build script to build all of our code and it calls the regular make commands in every directory. I was wondering if this script should fail on a failed test or if it should not care about unit tests. – tjwrona1992 Jul 20 '17 at 19:18
  • Do you complete the full build first and then run all of the tests afterwards? Or build and test each app one at a time? – tjwrona1992 Jul 20 '17 at 19:30
0

I say, if your unit tests take less than 1-2 seconds to run, do it. Otherwise don't. Building your application should never take long IMO, because you'll lose focus every time you have to build.

Your build should fail if your unit tests don't pass. That'll enforce quality. What good is a build where some functionality doesn't work as intended?

Plus, it prevents you from forgetting to run your unit tests. Or it prevents you from voluntarily skipping the tests on a friday 5o'clock build.

0

Yes. :D But there is no one-size-fits-all answer.

But, proper test automation always detects more issues earlier than ad-hoc unit-testing, saving you from larger issues further on. And, remember - if you're not building for release, you're building to test your latest changes, aren't you?

Depending on the size of your project, you should choose to run all unit-tests as part of the private build, if the added time is neglectable. But adding more than 10s to a 1 minute-build, or 2s to a 1 second-build is probably not a good idea. In that case, you might want to identify one or two quick unit-tests that makes a sanity-test of the most important functionality that you run on each private build.

However, as soon as you have a build-server that is able to pull from the VCS, you should also add the functionality to run "all" unit-tests whenever a new successful build is produced. Unless "all" takes too long time, that is... It boils down to the number of builds/day, average build-time and unit-test time. As the project grows, you will find that you will need to make the build modular enough that you don't need to unit-test unrelated parts of the code-base.

And no matter how clever you design your test-cases, you will inevitably find an important use-case that really needs to take a long time - be it processing intensive or simply needing real time to tick. So, for those cases you will also need conditional, or manual, invocation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.