At which point is a continuous integration server interesting?

Question

I've been reading a bit about CI servers like Jenkins and I'm wondering: at which point is it useful?

Because surely for a tiny project where you'd have only 5 classes and 10 unit tests, there's no real need.

Here we've got about 1500 unit tests and they pass (on old Core 2 Duo workstations) in about 90 seconds (because they're really testing "units" and hence are very fast). The rule we have is that we cannot commit code when a test fail.

So each developers launches all his tests to prevent regression.

Obviously, because all the developers always launch all the test we catch errors due to conflicting changes as soon as one developer pulls the change of another (when any).

It's still not very clear to me: should I set up a CI server like Jenkins? What would it bring?

Is it just useful for the speed gain? (not an issue in our case)

Is it useful because old builds can be recreated? (but we can do this to with Mercurial, by checking out old revs)

Basically I understand it can be useful but I fail to see exactly why.

Any explanation taking into account the points I raised above would be most welcome.

Answer 1

Is it just useful for the speed gain? (not an issue in our case)

Any time spent cycling your process is time you could have spent getting into the flow of developing.

You will also save time at milestones because you ideally are building fully packaged and ready-to-ship bits that can be burned straight to CD, uploaded to the web, etc.

Is it useful because old builds can be recreated? (but we can do this to with Mercurial, by checking out old revs)

No, you don't recreate builds. You use the build it created, and keep it around until your retention settings toss it out.

You build it on the build server, as opposed to on some Dev's box.

Here we've got about 1500 unit tests and they pass (on old Core 2 Duo workstations) in about 90 seconds (because they're really testing "units" and hence are very fast). The rule we have is that we cannot commit code when a test fail.

In addition wouldn't you like to be able to run automated integration or end-to-end tests on your code and catch issues that unit tests won't catch?

You wouldn't want to run those on dev boxes, because it would be a pain to set up and maintain that environment. Integration tests also tend to be very slow to execute.

at which point is it useful?

It is an investment like any other.

Use it once and you might come out behind or only break even. Use it on multiple projects and you will probably come out ahead.

It also depends on the type of applications you make.

If you make web applications that need to be deployed to a Java Application Server, or IIS, then CI becomes a no-brainer. Same if you have a database as part of your project. Deployment is painful to execute manually, and your QA team (if you have one) will need it as often as daily at some point.

Also, you might want to see how your needs and problems align with Joel Spolsky's 12 Steps to Better Code. Particularly 2, 3 and 10 (though they're all interesting and important).

Answer 2

At which point is it useful?

• Your build+test cycle gets over a couple of minutes. With 5 minute test run you will no longer want to run all tests yourself, especially for small changes.
• You start building multiple variants. If you have a couple of customers with different customizations, you should run the tests against each variant, so the amount of work will start growing quite fast. Than you'd run the test suite for one variant on developer machine and leave it to CI to run it on the rest.
• You write automated integration tests that need some non-trivial test environment setup. Than you want to test against one canonical test environment, since the developers may have their environment modified in various ways due to the development changes. The CI is most suitable place for the canonical environment.
• Testers can just pull the latest build from CI. Thus they neither need to have, learn and use the development tools nor has any developer have to send them their build manually.
• When you are preparing for release:
  • Testing becomes more important, so having one place with prepared builds for test is even more useful.
  • You are certain that all builds are built with the same build environment, so you avoid problems that may be introduced by subtle differences between developer installations.
  • You are certain that you are building exactly what is checked in the version control system. During development if somebody forgets to check something in, you'll find quite quickly, because it will fail for the next developer. But if such build slipped to QA or production and they report a bug, it would get very hard to trace.

You probably don't need CI just yet, but I think it will become useful when you get to the testing phase. Jenkins is set up in few hours and it will simplify your testing and help avoid silly mistakes (that happen especially when you hurry a quick fix to production).

Answer 3

For me a CI gets interesting if your team has more than 1 member.

You have to stop thinking of CI as "another PC running the tests for me". CI about having a defined and automated build process and release management.

CI is the single authoritative entity that creates your software release. If it doesn't build on the CI it just didn't happen.

With a CI you have restrictions to automate everything which will show you all the manual tweaks, hacks and shortcuts that you have in place and just don't work with a CI and should be avoided in the first place.

Problems it avoids:

• Who is responsible for building the release
• Is this person 24/7 available
• But it builds on my machine syndrome
• Removes all ambiguity about what was the version we released

Advantages (just too many to mention all):

• Automated version numbering
• Issue tracking integration
• Automated metric generation
• Static code analysis
• Automated deployment
• Multi-Stage setups

Answer 4

There's one fundamental problem about Continuous Integration (CI) that is perfectly mirrored in your question: CI practices are hard to implement and defend because CI server software is not trivial to setup, nor is it trivial to get your projects up and running through a CI server. With this, it becomes hard to actually see where's the gain in embracing CI at all.

First of all, CI is about insight and quality. Good CI brings you closer to your project, gives you proper feedback on quality metrics, documentation, coding standards compliance, etc. It should provide you with the tools to easily visualize all this, and allow you to at-a-glance recognize and easily associate a set of changes with a specific snapshot of all these project metrics.

It is not just about running unit tests. Not at all! Which brings me to quality. CI embraces errors, it does not avoid them or throw them away. What it does is quite simply provide you with a tool to error out early on, instead of later on. So you don't really commit previously tested code to a CI server. Although you should strive to commit clean and not broken code, you actually use the CI server to automatically run an integration builder automatically through your code and have it assess if everything came out right. If it has, neat! If it hasn't, no problem - good CI practices state that your next priority should just be to fix whatever has become broken. Which, in a good CI server, should be easily pointed out for you.

As a team's size increases, integration of everyone's code naturally becomes harder. It should be the task of a centralized CI server to test all integrated parts and take that burden off the team's members. So you must have everyone commiting early (and as cleanly as possible) and then monitoring builds status (there's usually notifications envolved). And again, if something gets broken because of some developer's commit, it immediatelly becomes his responsability to fix that and get those automated builds back to OK status immediately.

So you see, in my opinion every single project benefits from being Continuously Integrated. The thing is, until now and due to mind-boggling complexity from every single CI server I know of, people really fended off CI practices on smaller/simpler projects. Because, come on, people have better things to do than spending days configuring an ugly, overly-complex, under-delivering, bloated software.

I had this exact same problem, and that's what made me develop Cintient in my free time since about a year ago now. My premise was to make it simple to install, configure and use, and to make it deliver on those quality metrics that every one else pretty much fails or underdelivers. So, after this long answer comes my shameless plug of pointing out the GitHub link for the project (which is free and open-source, natch). It also has some nice screenshots, apparently. :-) https://github.com/matamouros/cintient

Hope I helped you.

(NOTE: Edited after Bryan Oakley's comment, on the fact that I should have taken more time to build a better answer. I also think he was right.)