What software models are appropriate for daily builds and continuous integration?

Question

On reading the joel test and about daily builds, a discussion with a few tech-lead friends of mine in various companies revealed that they never did daily builds or continuous integration because according to them:

1. Daily builds were meant for projects following Agile practices. It does not work for the waterfall model.
2. Continuous integration required automated testing which their OpenGL/OGRE/OSG type-of graphics project couldn't use, because apparently automated testing is impractical for projects where the graphics part is under (re)development.
3. One of them believes that it's not necessary to have configuration management. It's just enough to follow the philosophy of it, by defining small 2-3 day tasks, perform all activities of the development cycle for each task (code, code-review, unit test, integration), deliver to integration stream and build and generate the setup.
4. Even if they created scripts for the entire build and burn-to-cd process, creating automated tests were a problem because in any program you create, you can't always know what tests you'd have to run and scripting a test case for it takes up too much time and automated testing tools may not always support the specific kind of tests you might want to make.

Is daily builds and continuous integration really practical for non-agile projects? Is it meant only for test driven development? And is it really sufficient to follow the point 3 (above) kind of philosophy? I was talking to them about the build-in-one-step kind of scripted automation that Joel talks about, but they weren't willing to buy the idea.

p.s: I've been through the questions on this site about daily builds etc, but I believe this question is different.

EDIT: Point 3 should've started with "One of them believes it isn't necessary to integrate configuration mangaement with development tools"

Answer 1

Your friends are being lazy and IMO unprofessional - the world will soon pass them by.

Daily builds were meant for projects following Agile practices. It does not work for the waterfall model.

I don't have a definitive reference handy, but daily builds were common at least as far back in the 90's, so they pre-date Agile, and the *Unified Process timeline. Any time you have more than a couple of devs, SOMEONE will check in something which can be tested, and at the same time can now automate checks for breaking the build.

automated testing is impractical for projects where the graphics part ...

Automated testing is a time and cost saving tool. Why would anyone want to make software more expensive than necessary?

not necessary to have configuration management

That is plain crazy. The dev's all use different versions of 3rd party libraries? No one tracks which features are to be deployed in each release? No branch capability in Version Control System?

Answer 2

To be fair to your discussion partners, if what they have works, then, well, it works. I'd still probably sell them CI in some form, but I can understand the resistance.

There is an overhead to getting CI set up in the first place, and an overhead for managing it. Not a huge over head, but I point blank refuse to be "The CI guy" ever because I know at some point it will become an unwanted chore.

However, once I've nominated someone else to get it configured, at the very least it is beneficial to know that the code checked in is compiling. That one benefit alone is significant.

The testing side of it is kind of irrelevant here. The CI build should run tests, but those tests shouldn't exist if they serve no useful purpose. There are always test cases that do make sense though. This is also a point of resistance for many developers.

Some people still see code test coverage as a golden statistic of project health. It isn't. I'd still rather have 1pc coverage with that 1pc being appropriate, quality tests, rather than 80pc coverage of "Tests I wrote because we needed to get to 80pc coverage".

And that's the final issue. Once you've got CI, someone is going to ask you for coverage statistics, then they will ask why your coverage isn't higher, then someone will send you a link to a great UI testing package that automates... In their mind they've gone from a process which demonstrably works, to a process which requires more effort to get to the same end result, the same working code they were delivering previously.

Answer 3

Daily builds were meant for projects following Agile practices. It does not work for the waterfall model.

I would say: pure waterfall does not work for more than trivial projects with a lifetime so small that setting up a daily build would be oversized. But if your project lifetime is at least several months, and you mean "waterfall with recurring cycles", then of course daily builds can make a lot of sense. And even in a classical waterfall, where you have an analysis, a design and a development phase, in the development phase you have to do a lot of builds and code integration, so why not automate this and make sure the code changes integrate daily?

Continuous integration required automated testing

Nonsense, CI can integrate automated testing, but that's not a requirement. You can still test manually and use CI only for making sure you have a daily build ready for your testers.

One of them believes that it's not necessary to have configuration management.

If you have a project where the life cycle of your product ends after release 1.0, then this is true. In all other cases, you should have some kind of config management. But this is independently from CI. Furthermore, CI encourages you to keep your changes to your code base small and reintegrate daily - which exactly was your argument here against it.

Even if they created scripts for the entire build and burn-to-cd process, creating automated tests were a problem

See my comment, CI does not require automatic testing.

And to your other points:

you can't always know what tests you'd have to run

all of them, what else?

scripting a test case for it takes up too much time

You should script those test cases where applying them manually over and over again takes too much time, so having them automated will save you time.

automated testing tools may not always support the specific kind of tests you might want to make

Because there are some possible tests you have problems to automate, you don't automate anything? Does not sound very convincing to me. And who told you automated testing needs special tools? I would just start without any tools or only with some lighweight xUnit framework.

Answer 4

RE: Testing difficult-to-specify & difficult-to-test systems such as graphics systems.

In my experience, testing is not just (even?) about testing the output of the system that you are building. It is about verifying that the assumptions that you have made are correct, and continuing to verify those assumptions as the project evolves and situations change.

To this end, the placement of a moderate number of assertion statements at various points in the code has enormous benefit when combined with a continuous integration regime.

The system does not even need to be exercised with a comprehensive suite of unit tests (although these are a good thing if you have the time), Even a small set of vigorous system-level exercises will provide tremendous utility in uncovering bugs and building confidence.

Answer 5

Wow, where to begin? My personal opinion is that the friends you were talking with have a relatively narrow experience base and/or definitions. I see it every day when select other engineers claim I can't be really working on SCADA because I'm not working in Oil and Gas. It's an overly narrow definition which they carry with the conviction of religion and it leads them astray.

1. In 25 years of critical infrastructure systems development, delivery and support essentially every product development or delivery project I've been on has officially been a "waterfall" project. Some industries are simply that way. As a practitioner though: I set small goals based on the most important missing functionality, create tests to demonstrate delivery of it and then write the code to pass those tests. Sometimes the code writing drives additional tests (usually for all the error handling) and I normally try to create and test the error handling before the actual functionality because it's easier to expand the error traps than it is to add them after the fact.

A regular system build has always been part of the work when there's been active development and one of my early career responsibilities was starting the system build Friday about 5pm and checking it through to completion about 24hrs later. (Sunday was used to re-build where necessary after things went sideways on Friday night.) Coder compilation during the week was restricted to individual processes or sub-systems with mandatory check-in by noon Friday so that the system build could start on time. Quite honestly, the team and project leads would have loved to have a daily build available to them so the idea that a daily build is only applicable to "Agile" methodologies strikes me as underwhelming.

2. Automated testing of graphics based UIs is a problem area. If the UIs' underlying technology doesn't expose enough of the data structure and controls to allow addressing the display elements by name and reading the values programatically then you're stuck with screen scraping and any minor tweak for usability moves or re-formats information and breaks the screen scraping step which is the input to the UI test automation.

Personal opinion: differentiate between automated processing and UI in your test plans and focus automated testing on proving the automated processing. Let people do the graphical recognition thing that they're good at. Forcing the machine to do graphical recognition and insisting the only possible testing is the UI ignores the realities of software design and development even more than usual. (Engineers are already bad at writing tests, we shouldn't be so foolish as to forbid ourselves the workable tools!)

3. Configuration management can be achieved by a combination of means. My current system builds take 6 to 8 hours to complete and deploy on top of an installed system in 5 to 10 minutes. The deployment doesn't check that every one of 90+ software line items including mandatory third party products is correct because we manage those through a different mechanism and getting builds' installation tools to also install all of those items would be extremely costly. Instead, many of the line items aren't changed by the system build and we install them separately during server creation. It's not perfect but controlling critical infrastructure code includes blocking all the automatic self-updating features and being responsible to plan and conduct updates on every line item of software. It does at least let us handle each line item in an appropriate update cycle and validate the upgraded combination. for example: functionality which is controlled by data modelling gets updated much more frequently than code and application code is updated more frequently than OS version. three different tools are appropriate so that each one is handled appropriately for itself. So: CM is mandatory but that doesn't mean you must put every item in one tool. Learn and understand the life cycles involved and assign each part to an appropriate method and team.

4. Scripting test cases is an economically provable point. What's the cost of manual regression testing for each customer and release? I often see test suites that require several man-years to run on software systems that must be patched every 30 or 90 days. What's the cost of a bug being found by the users? How often does each happen? Very, very quickly you can come up with an amazingly large dollar figure as the cost of manual testing or not testing. Of course, if you're forced into screen scraping then the cost of tests could also get quite large very quickly so you need to identify ways to test that provide value at an acceptable cost and focus on implementing them. If the customer or product manager aren't willing to acquire and apply a suitable toolset, then you're forced into planning test cases to take the least amount of time and compute resources. Just don't skip formalizing your test cases because that is a false saving which will bite you eventually.

Which doesn't address the problem of matching tests to the appropriate release. Fortunately, that problem is already addressed for us in our version control system. One system I recently worked on for CM required a unique deployment for each server role and we hadn't gotten to fully generalizing the deployment steps so the procedure needed to be partially unique for each release. The direct solution was simply to include the scripting in the release and in version control. As each release was finalized, the scripts got touched to install that specific version of the build and the build testing could include the deployment on the test server.

For a lot of testing, you can do something similar: bundle the test scripts in with the product release kit. Testing as part of the CI pipeline then becomes compile and link, package, deploy, run the test shell.

My experience has been quite different from your friends' as evidenced by my opinions being almost opposite theirs. CI can easily be part of a waterfall project just as individual practices can be agile within each step of the waterfall. Using the philosophy is very practical because it helps you deliver highly reliable work products in a reasonably predictable amount of time and protects the quality of the final product from regressions. Even in the large engineered systems which make up critical infrastructure, the required functionality can change mid-project and users change their minds about what they need which forces the project to go into change orders and scope negotiations. Handle change poorly and the project is in the weeds rather than on track. Agile practices are about handling change as effectively as possible for the system and technology in use and your selection of which ones to apply has to make sense for the project size and organizations' capability maturity.