34

There is an overhead associated with continuous integration, e.g., set up, re-training, awareness activities, stoppage to fix "bugs" that turn out to be data issues, enforced separation of concerns programming styles, etc.

At what point does continuous integration pay for itself?

EDIT: These were my findings

The set-up was CruiseControl.Net with Nant, reading from VSS or TFS.

Here are a few reasons for failure, which have nothing to do with the setup:

Cost of investigation: The time spent investigating whether a red light is due a genuine logical inconsistency in the code, data quality, or another source such as an infrastructure problem (e.g., a network issue, a timeout reading from source control, third party server is down, etc., etc.)

Political costs over infrastructure: I considered performing an "infrastructure" check for each method in the test run. I had no solution to the timeout except to replace the build server. Red tape got in the way and there was no server replacement.

Cost of fixing unit tests: A red light due to a data quality issue could be an indicator of a badly written unit test. So, data dependent unit tests were re-written to reduce the likelihood of a red light due to bad data. In many cases, necessary data was inserted into the test environment to be able to accurately run its unit tests. It makes sense to say that by making the data more robust then the test becomes more robust if it is dependent on this data. Of course, this worked well!

Cost of coverage, i.e., writing unit tests for already existing code: There was the problem of unit test coverage. There were thousands of methods that had no unit tests. So, a sizeable amount of man days would be needed to create those. As this would be too difficult to provide a business case, it was decided that unit tests would be used for any new public method going forward. Those that did not have a unit test were termed 'potentially infra red'. An intestesting point here is that static methods were a moot point in how it would be possible to uniquely determine how a specific static method had failed.

Cost of bespoke releases: Nant scripts only go so far. They are not that useful for, say, CMS dependent builds for EPiServer, CMS, or any UI oriented database deployment.

These are the types of issues that occured on the build server for hourly test runs and overnight QA builds. I entertain that these to be unnecessary as a build master can perform these tasks manually at the time of release, esp., with a one man band and a small build. So, single step builds have not justified use of CI in my experience. What about the more complex, multistep builds? These can be a pain to build, especially without a Nant script. So, even having created one, these were no more successful. The costs of fixing the red light issues outweighed the benefits. Eventually, developers lost interest and questioned the validity of the red light.

Having given it a fair try, I believe that CI is expensive and there is a lot of working around the edges instead of just getting the job done. It's more cost effective to employ experienced developers who do not make a mess of large projects than introduce and maintain an alarm system.

This is the case even if those developers leave. It doesn't matter if a good developer leaves because processes that he follows would ensure that he writes requirement specs, design specs, sticks to the coding guidelines, and comments his code so that it is readable. All this is reviewed. If this is not happening then his team leader is not doing his job, which should be picked up by his manager and so on.

For CI to work, it is not enough to just write unit tests, attempt to maintain full coverage, and ensure a working infrastructure for sizable systems.

The bottom line: One might question whether fixing as many bugs before release is even desirable from a business prespective. CI involves a lot of work to capture a handful of bugs that the customer could identify in UAT or the company could get paid for fixing as part of a client service agreement when the warranty period expires anyway.

9
  • 13
    It can be beneficial even for a team of one. Especially if you've got a long running test suite, it is much better to get automatically results of an overnight build and test run than doing it manually all the time.
    – SK-logic
    Apr 2, 2011 at 10:01
  • 3
    @Carnotaurus, local clone of remote git repository gets rid of the timeout from source control. Network issues - for building the product? Now, really...
    – user1249
    Apr 3, 2011 at 8:12
  • 3
    @Carnotaurus red light due to data quality or infrastructure is a codesmell of Fragile Tests. See also so: managing-the-maintenance-burden-of-unit-tests
    – k3b
    Apr 3, 2011 at 8:59
  • 1
    The more a test depends on external aspects the more fragile it is. Example: If a test only succeeds if customer XY is in database the test if fragile unless the test-setup makes shure that this prerequisit exists by inserting the data itself if neccessary.
    – k3b
    Apr 3, 2011 at 9:22
  • 6
    "Bottom line: Why make working products when we can get someone else to pay us for fixing it, because it isn't like they expected the software to work in the first place"? It may not meet the legal definition, but that sounds like fraud to me. Jan 2, 2012 at 17:35

12 Answers 12

43

Setting up a CI-engine is akin to setting up a fire alarm in a house.

In my mind the benefits do not correlate with many developers, but with a large code base. The CI-engine actively do all the boring work you do not want to do yourself, and do it everytime.

If you break a remote module which you haven't touched for a long time, you are told immediately. Not only compilation wise, but also functionwise if you have set up unit tests.

Also note that if you let you CI-engine do all the boring work, including setting up installers etc, you do not have to do it manually. You can just check in your source, and await the finished product being built in the standard location. (EDIT: The CI-engine also works in a well-defined environment, avoiding any developer specific setups, ensuring reproducibility)

This is also a part of quality assurance.


EDIT: After writing the above, I have had experience with the Maven build tool for Java. Essentially this allows us to keep the complete CI-configuration inside the project (with pom.xml) making it much simpler to maintain the CI-installation and/or migrate to another CI-engine.

15
  • 1
    @Carnotaurus, in that case the red light is being used for transient errors too. Sounds like a limitation in the CI-engine you are using.
    – user1249
    Apr 3, 2011 at 7:49
  • 2
    @Carnotaurus in my experience the work done to thoroughly document every aspect of everything regarding e.g. doing a release and then doing said releases a number of times, is larger than capturing the workflow in an ant or maven script which can then be executed by a robot any number of times. Said robot also works in a clean room environment ensuring that builds are reproducible.
    – user1249
    Jan 1, 2012 at 11:37
  • 1
    Note that the breaking I am looking for is not failing unit tests but that the actual programs we ship can still build. It is not as important anymore as we migrated to maven artifacts instead of compiling everything for each release, but it is still important to know that the build is fully functional. We just need the Continuous Deployment part too.
    – user1249
    Jan 1, 2012 at 11:50
  • 2
    @Carnotaurus: I am not talking about an alarm system. If the tests fail, the patch is not integrated into the main repository (at all), ensuring that whenever you checkout/clone you get a fully working copy and can start working right away. Now, I agree with you that tests can be difficult to write and maintain... but I have worked with and without test, and I see a definite quality improvement with them. So the question is: automated or not ? In my experience, it takes about as long to test manually than to write the automation script (but I work in a large corporation with tools for that). Jan 1, 2012 at 12:20
  • 5
    "It's a lot of work to capture a handful of bugs that the company would get paid for fixing as part of a client service agreement anyway" - well in that case you have economic incentive NOT to produce software with as few bugs as possible, and in that case all this does not apply to you.
    – user1249
    Jan 7, 2012 at 11:44
33

It's not how many developers, but how many steps it takes to get from 1 to n (inclusive), where 1 & n are ...

1: Checking out code
And
n: having installable\deployable packages

If n < 2 you maybe don't need CI
otherwise, you need CI

Update
From reading your findings I can only conclude that you approached CI from the wrong direction and for the wrong reasons.

13
  • 1
    +1 - I've quite a lot I could add to the above - but this is very close to the core of why I like CI... not only for what it does but also for what it requires you to do to make it work (those requirements being things you really ought to do anyway).
    – Murph
    Apr 2, 2011 at 12:18
  • 2
    @murph: Yup, and it brings little advantages like 1) knowing what's in your repository will build, 2) single click build, 3) being able to release a version at a moments notice and So much more Apr 2, 2011 at 12:51
  • So it's fair to say that it depends on the complexity of what is to be released, measured by its number of steps to deployment as being able to "test" separate layers?
    – Phil C
    Apr 3, 2011 at 7:28
  • 1
    @Carnotaurus: Sorry mate, but I'm not sure I know what you're trying to say. CI has nothing to do with testing. Yes, you can - and should - run unit tests as part of the build, but they shouldn't be dependant on anything that isn't setup as part of the test. However CI does aid testing. One of the - many - benefits of CI is that it allows a QA team to immediately & seemlessly pickup new code changes. CI = The Ability To Release Immediately Apr 6, 2011 at 6:50
  • 1
    @BinaryWorrier - I think step 1 is checking in the code ;)
    – user23157
    Feb 4, 2012 at 12:16
10

It can be worth the effort even for a team of one. This is especially true when you are developing cross platform code and you need to ensure that your changes will work on both platforms. For example, Microsoft's C++ compiler is more accepting than GCC, so if you develop in Windows but need to support Linux as well, having a CI system tell you when your build breaks on Linux is a huge help.

Some CI systems are quite easy to set up, so the overhead is not really that huge. Try Jenkins or Hudson for instance.

1
  • I had not considered a cross-platform scenario. If different compilers are needed to create different builds then there is a strong case for CI - take an upvote.
    – Phil C
    Apr 3, 2011 at 7:49
4

As you say there is an overhead cost of setting it up and keeping it running.

But the question of where is the break even point is not a function of how many people you have in your team, but rather a function of the length of your project.

Having said that, there is a part of the setup cost that you can use in all your future projects, so in the long term the overhead cost may approach zero.

4
  • So length of the project (size of project) is important for you CI? I have found the false alarms to be very costly.
    – Phil C
    Apr 3, 2011 at 7:37
  • Yes, you need to have the time to pay back the setup and learning curve costs. In theory you should over time learn how to eliminate the false alarms. Apr 3, 2011 at 9:39
  • Yes, this is theory
    – Phil C
    Apr 5, 2011 at 16:54
  • Well, no, its practice. Over time, you make a note of what tests are fragile and what tests aren't, and you don't worry too much if your fragile tests break unless they break several times in a row. You write more isolated, robust tests as you learn how, and you let legacy coverage build over time instead of doing it all at once. In practice, CI is a not a silver bullet, its a process change that takes time and eventually leads to less buggy software.
    – philosodad
    May 22, 2012 at 13:45
3

I set up Jenkins this week to build a small .NET project I'm working on. I integrated it with my Git source control so that it triggered a build on every commit. I integrated the unit tests into the build. I integrated static analysis in the form of FxCop and StyleCop violations.

Now every time I check in, it checks out all of my code, builds it, increments the version number across all assemblies, tests it, analyses it for FxCop and StyleCop violations, archives the build and records the results in a number of graphs so I have visibility over time of test results and violations.

Doing that from scratch takes an hour or so (maybe a day with Google if you've not done it before). It costs nothing because all the tools are available for free.

If, as and when the project builds I have a high quality infrastructure that will grow with it at no cost. If or when new developers join the project I can get total visibility on their work and their impact on the project for no cost.

So the only scenario I can see CI not being worth while is either for a project that will take a day or so and then never get revisited, or one in a language where there are no existing / free tools available and the cost of acquiring them is disproportional to the work.

2
  • As I said, it is the unit tests to cover legacy code that presents a major cost. Also, we are not talking about a one man team here either... I appreciate the heads up on Jenkins though.
    – Phil C
    Feb 4, 2012 at 12:16
  • 1
    For what its worth, I gained considerable benefit from running a CI server without having any tests - just from the confidence in clean builds and the availability of deployment packages.
    – Murph
    Mar 24, 2012 at 14:14
1

If you can verify all aspects of your project after each change, then you don't need CI.

In all other cases, it's a net win.

2
  • I think this is where I'm coming from too. It's much cheaper too.
    – Phil C
    Apr 3, 2011 at 7:45
  • What do you mean by verification in this case? If it is automated, happens as frequently as possible, and helps identify mental slips & oversights, then I am all for it. :-) Feb 6, 2012 at 14:28
1

Overhead is minimal. I'd say for one man projects it would be useful. Once you reach two it's invaluable.

I agree that for one man projects if you have a "one step build/verify" then you might be okay with continuous integration, but in those cases you've done most of the hard work to set up CI so might as well just put it in a formal system (CC, Hudson, TeamCity, etc).

1
  • You mean without CI?
    – Phil C
    Apr 3, 2011 at 7:38
1

The question of, when does CI pay for itself, is a hard one to answer on a new project.

On an existing project, it's much easier to see. If you find a critical bug at the development piece of the supply chain, you know that problem exists at the earliest point possible. The cost for fixing that is now the lowest. If that problem exists in any environments above development, your cost is higher.

Now, management might decide to release with a bug, but that's a business risk. At least now it can be mitigated through that risk assessment, rather than late night phone calls/panics, which, if they were billed at hourly rates, end up being mighty expensive.

Another thing to consider in terms of cost. What is your QA department like? Is it manual testing? If you go CI, you may be able to reduce overall QA costs by automating those scripts against your dev box. You may be able to reduce the number of QA folks supporting your project by getting them involved in the CI phase.

5
  • 1
    To replace a manual regression test with some UI automated tests takes quite a bit of time and skill. In one company, one chap wrote about 40% of the tests and left the company. Several years on, the tests still have no been written. I bet this is typical.
    – Phil C
    Apr 3, 2011 at 7:44
  • No, it's not typical.
    – quant_dev
    Apr 3, 2011 at 9:48
  • I haven't seen much counter-evidence
    – Phil C
    Apr 5, 2011 at 16:55
  • If you write integration/acceptance tests in natural language DSL like Gherkin, you can easily translate the automated test to manual. So I don't see that as a problem. Apr 6, 2011 at 13:33
  • @Carnotaurus - I think it is typical. I've seen it twice too. UI automated tests are hard.
    – Rocklan
    May 10, 2013 at 2:13
1

CI is always always always worth it: The sense of security it gives you enables you to work at a faster rate than would be possible otherwise. The problem that you have seems to revolve around unit tests. I agree that unit tests are very expensive, but also think that (for many things) they are the least worst option that we have. Personally, and for the sorts of systems that I tend to work on, I swear by system-level tests operating on a combination of real-world and (possibly synthetic) pathological cases; It is cheaper than unit tests, and gets into those hard-to-reach corners of your conceptual universe: Donald Rumsfeld's "Unknown Unknowns".

5
  • I like the bit about system-level UI tests. Much of testing gets lost in a fog of unit tests of questionable quality and coverage.
    – Phil C
    Jan 6, 2012 at 15:02
  • Coverage is a question of human psychology also; we have an innate tendency to write tests for those cases that we have already thought about. It is for this reason that to be certified at a high SIL level, unit tests need to be written by a separate team from that which developed the system under test, and even then, there are many cases & situations that are obvious to everybody only in retrospect. Rigorous checking of code coverage is needed; but extraordinarily difficult and expensive. Jan 7, 2012 at 0:01
  • ... Hence the benefit that you get by throwing the nastiest junk that you can find into the system at the top level and seeing what happens... :-) Jan 7, 2012 at 0:02
  • 1
    +1 - Totally agree. Some parts of a system are just not unit testable (i.e. the edges like at the database). Sure you can mock the db, but that leaves the code that talks to the db untested. At some point you need to write some real world tests that integrate your systems and talk to other systems. When you do that you always find bugs that you missed in the cosy clean world of the unit test and the mocking framework (LINQ to SQL springs to mind).
    – user23157
    Feb 4, 2012 at 12:13
  • Actually, I recently discovered an interesting take on fuzz testing that might redeem unit testing: jester.sourceforge.net Mar 7, 2012 at 14:31
0

Always use it, regardless of the size of the team. If it's just you for example, who knows, you might be coding from your laptop at starbucks then continue from your beefier system at home?

The overhead really isn't that bad.

0

One. Yes, one is enough to start using continuous integration.

0

It is not a matter of how many developers, but of

a. How much time you save using automation and how often.

  • Time saved on building after integrations, running automated tests and creating setups * frequency of check-in = percent of time saved.

b. How many versions/branches/products you have.

  • If you have one developer working on two different branches then the time saved is doubled, since each branch would require building, testing and packaging.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.