6

We are trying to adopt some CI practices and methodologies into our organization. I'm currently reading over some of the points in "Continuous Delivery" book written by Jez Humble and David Farley. And the point that I am debating is when unit tests should be run - before or after git commits. The book seems to suggest running unit tests after you check in, as a part of a commit test suite. But wouldn't it make more sense to conduct unit tests before checking in?

Any comments would be appreciated.

10

Usually the developer runs them manually before checking in, and a CI server runs them automatically after checking in. Programmers are usually pretty good about running unit tests for incremental builds on the configuration they've been working on, but they don't do things like do a clean build from a completely clean checkout from version control, run tests for all configurations of a product, run all the unit tests for the downstream projects that depend on theirs, etc.

That's why CI servers "rerun" the tests after commit. You can configure them to run the tests before they actually get merged into a master branch, if you want a branch that definitely has had all the tests run.

  • The more interesting problem is: what happens when there are not enough resources to run all the tests than we want to run for every commit? This is likely the case in all big projects. – Ciro Santilli 新疆改造中心法轮功六四事件 Dec 14 '17 at 17:11
2

Running your automated tests on your CI Server is actually a form of Integration testing.

When a developer runs tests locally they are validating whether the changes they have made mesh with the state of the entire system when they checked out their code.

Your CI Server, on the other hand, is continually integrating work from all developers. When it runs automated tests it's to validate that developer changes mesh together correctly.

0

Committing broken tests / broken code to your local repo is not problematic in and of itself. Problems start to arise when one locally commits to a specially-designated integration branch in anticipation of a push to a central repo. Real problems do arise when one pushes broken stuff to a specially-designated branch on a central repository.


A system that completely precludes committing broken tests is a broken system. If you follow the mantra of writing tests before writing code, the tests will necessarily fail at first. Making it impossible to commit those intentionally broken tests does not make sense.

A system that completely precludes committing broken code is also a broken system. Developers may start with pseudo code that doesn't even compile, then proceed to real code that doesn't work, and finally to real code that apparently works. Making it impossible to commit those initial ideas does not make sense. Having my broken code committed to my local repo has saved my rear end more than once.

Stealing words from the Zen of Python, "Configuration management is one honking great idea -- let's do more of those!"


I tend to work in esoteric environments where complete local testing is impossible (e.g., supercomputers, networks of powerful non-supercomputers, embedded devices, and networks of embedded devices, none of which I have on my desk). This makes me a bit overly leery of overly aggressive hooks in a CM system.

Even if you don't work in such esoteric environments, having a commit hook that automatically triggers testing for every single commit (and then rejects the commit in the case of a failure) is just a bad idea. It's best to encourage people to commit frequently and to merge frequently. Merging in a branch that contains uncommitted code is a recipe for a good amount of configuration management angst. Failing to merge frequently is a recipe for even more configuration management angst.

The opposite extreme, having no hooks at all and then making developers do embarrassing stuff when they break the build, is also a bad idea. What's needed is a compromise that empowers people to be their most efficient and yet protects against people doing the stupid things that even very smart people are wont to do.

  • Commit broken code to your personal dev branch all you like, but make sure it works and squash it before pushing it into the master repository. The last thing I need is to think I reverted a bad change only to end up with a version that doesn't even compile. (End rant) – RubberDuck Jun 18 '16 at 0:25
  • 1
    @RubberDuck -- Sometimes you need to share broken code with other developers. I'm leery of making repos on my personal machine visible to all. Git has branches! Pushing garbage to an in-progress development branch on the central repo should have minimal consequences. Pushing garbage to an integration branch, or even worse, to the master branch on the central repo might well have big consequences. Those special branches on the central repo should always compile and should run. While that's not always the case, that is the goal. – David Hammen Jun 18 '16 at 0:33
  • Then we completely agree @DavidHammen, but that's not clear from your answer. – RubberDuck Jun 18 '16 at 0:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.