7

I've had a dispute with a colleague about when to run JavaScript unit tests in a CI environment at one of two different times. Let's call the two parties PE (pro-early) and PL (pro-late).

Early (against src)

Running the tests against the source code right out of the version control system.

PE says that we should run the riskiest code first and "fail fast." Both parties agree that we should run unit tests this way during development because the cycle time is much faster. PE says that since we need to have the build configured to run early in any case, an alternate configuration that runs the tests late constitues duplication and bloat.

Late (against dist)

Running the same tests against code has been packaged up (concatenated and minified).

PL says that we should test the code as it ships. PL claims to have experienced time-consuming scenarios where the code worked when run directly from source, but failed because of something that went wrong when the code was packaged for distribution.

PE says a hypothetical "something might go wrong" is a weak argument. And anyway it's not CI's job to catch that kind of problem. The distributable code is tested by QA before it's shipped.

Further details

"Early and late in CI" is not an option, much to PEL's dismay.

The code is modularized using AMD and RequireJS. We're using Karma for unit tests and Grunt for the build script.

So which is it -- early or late?

Is one side definitely right? Are we missing any considerations that may sway one person to the other side?

  • Sounds like you've already answered your own question. Both parties agree that you should run unit tests during development because the cycle time is much faster, there is no option to do it both ways, and the distributable code is tested by QA before it's shipped, so the correct course of action seems clearly apparent to me. – Robert Harvey Sep 8 '15 at 20:43
  • Running both early and late is absolutely an option in development. It's only in CI where have to choose one or the other. (The reason is silly, and we'll get past it, but in the mean time we have to choose one or the other.) – Patrick McElhaney Sep 8 '15 at 20:47
  • Left out some details in order to make the question more generic (not "too localized"). I'll share them now. The issue is when running tests against source, Karma tries to open to many files at the same time. Running "early" in CI is currently impossible. So I changed the build script to run the tests "late". Someone complained, said we should run "early" or not at all. Was trying to explain why running the tests "late" is a good thing to do regardless. – Patrick McElhaney Sep 10 '15 at 13:18
8

From a generic testing point of view:

"Testing early" is a workflow optimization. It helps you iterate faster by failing quickly. Tests that are close to source code, like unit tests, are useful here.

"Testing late" is a requirement for quality assurance. You must test code exactly as it ships, because as you said, there are multiple places where things can go wrong even after the source code is validated. High-level tests that integrate multiple components are useful here.

You should be doing both low-level and high-level tests. I'm not sure why "early and late" is not an option, but that is exactly what I'd recommend.

  • 1
    The inability to have both "early" and "late" will be resolved eventually. It involves waiting for a ticket to make its way through Jira. In the mean time we're stuck choosing one or the other. – Patrick McElhaney Sep 8 '15 at 20:53
  • 4
    @PatrickMcElhaney: I don't know what this has to do with Jira, but honestly, if your are a slave of your issue tracker, you have far bigger problems than your current question. – Doc Brown Sep 9 '15 at 8:08
  • @DocBrown See my comment on the question. Yes, I have "bigger" problems, and I'm surrounded by smart people who are working various problems of various sizes. I'm focusing my energy on the problems where my I'm able to make the most impact, given my unique combination of position, skills, and interests. – Patrick McElhaney Sep 10 '15 at 13:27
1

Typically you'd run unit tests early - they're developer tests that the bits you've developed work as expected in isolation. Then you'd run wider integration tests late, as these test that the bits you've developed work as expected with the other bits.

There's no need to run unit tests both early and late - its redundant to run them late if you already run them early, but you can never run integration tests early (obviously). CI is the place to run these.

1

As a general case, every change to the code should be tested. Exactly how you do those tests and what tests are done is specific to the situation and environment you find yourself in.

Testing both early and late would be the recommended alternative. You say this will eventually be the case, but until then you need to pick one over the other.

It really just sounds like you are looking for the best compromise until the situation matures and allows for both. Pick the one that tests the highest risk change and then allows for the fastest corrections to the code - by the sounds of it, this is the test early for now. If the QA is done after all of this, it gives you one last opportunity to catch problems.

As already mentioned, the testing early with unit tests identifies logical errors and helps correct those before heading into integration. Once into integration, the testing late identifies integration and compatibility issues. It also offers final system testing and user acceptance testing.

Given that you are using CI, it is most likely a highly automated environment anyway, use that, automate as many of the tests as possible. This gives you a higher level of confidence you are not introducing regression issues. I'm glad you have identified the source of why you can't do both, the ticketing system should support the speedy delivery of quality code, not inhibit it.

0

Thanks for the responses. I'm surprised at the extent to which people think that "failing fast" is important within CI. There's a difference between the development (individual) cycle and the integration (team) cycle.

Development

A developer makes a small change. Runs the tests to ensure that change didn't break anything. If something did break, the developer inspects the code, notices an error, fixes it, and runs the tests again. That cycle can literally happen a couple of times a minute, provided the tests are able to run almost instantly. If it takes a few minutes for the tests to run that will blow out the cycle time and disrupt the developer's workflow.

Integration

A developer merges a change. The build runs and something fails. The entire team is notified of the failure. The team communicates to figure out who will work on the problem. A couple of developers put their heads together, pull down the latest code, run the build locally, figure out what went wrong, make a change, commit the code with a thoughtful commit message, push the code up to the server, and merge. There may be other time-consuming steps. That's generally the bare minimum.

In this case, if it takes several minutes longer for a failure to be discovered (the difference between "early" and "late") it doesn't have much impact on the overall cycle time.

If you choose not run tests against the final product during CI, and something about the concatenation / minification process causes a bug to be introduced, it may be hours, days, or even months before that bug is eventually discovered.

Running the tests "early" in CI is a "nice to have." The impact on overall cycle time is marginal. Running the tests "late" is a "must have". Failing to do so can be catastrophic.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.