Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
In a meeting the other day a claim was made that agile was only 60% as efficient in development time when compared to waterfall. I am not looking to validate or refute this claim. I am interesting in finding out if there have been any studies comparing the 2 methodologies.
Are there any studies out there comparing the two?
The book "Making Software: What Really Works and Why we Believe It" has some chapters on Agile methods including XP, Scrum, Dynamic Software Development, and Lean, with good scientific backing. It's high quality, as you would expect from O'Reilly. One of the editors was the excellent Greg Wilson, a trusted computer science author, editor and presenter.
The book itself summarizes multiple research studies, including many on agile. One section summarizes research including "Are Two Heads Better Than One? On the Effectiveness of Pair Programming" by Dybå, T.; Arisholm, E.; Sjøberg, D.I.K.; Hannay, J.E.; Shull, F.; and "Empirical Studies of agile software development: A systematic review" by Tore Dybå and Torgeir Dingsøyr.
The general sense is that most agile practices are beneficial, but that the effects of pair programming and TDD and other agile tenants are not as strong as one might hope. There's even a disturbing footnote that TDD may in fact be somewhat addictive*.
The book is a great way to get access to a lot of research that's been done, all in one cohesive whole. There are a few blogs and other sites on the web that review the book.
*This isn't necessarily my opinion!
As much as I dislike the title, I believe that Balancing Agility and Discipline: A Guide for the Perplexed might contain some information that is relevant to you. This book by two software engineering process and software project management experts - Barry Boehm and Richard Turner. This book looks at various aspects of the agile and plan-driven methodologies, compares and contrasts them, and also discusses integrating them to achieve a "best of both worlds" situation.
Appendix E of Balancing Agility and Discipline contains a wealth of empirical information regarding the costs and benefits of various agile and plan-driven methods. However, there doesn't appear to be any data regarding time effectiveness. But glancing through the data, it appears (as I suspected) that this isn't an either/or choice - some projects experienced decreased effort, faster schedules, and lower defects when applying agile methods. However, other projects that used. The section discusses a number of different projects in different industries, the type of process they used, and what they experienced over the course of the project.
There are plenty of case studies cited in Appendix E that yields this data. There are far too many for me to just start naming randomly, since many are focused in a particular industry or even within a particular organization. If you are going to look at cases, I would suggest finding those that are similar in nature to your team, project, organization, and industry to draw reasonably valid conclusions.
In Rapid Development: Taming Wild Software Schedules, Steve McConnell identifies a number of factors to consider when choosing a lifecycle methodology: level of understanding of the requirements, level of understanding of the architecture, desired reliability, risk management, schedule constraints, amount of process overhead, mid-project "course corrections", ability to provide the customer with visibility, ability to provide management with visibility, and sophistication of the development team and management. There are others, as well, such as organizational culture, so there probably isn't an exhaustive list anywhere.
Even given the exact same project, there is also the team factor. If you take a team that has consistantly delivered software using the plan-driven spiral methodology and throw them into Scrum, they are going to experience a decrease in productivity, an increase in thrashing, and have to overcome a new process model before they can come around to being successful. Even though another methodology might be more suited, there's always the business need to actually deliver the software. That's why process improvement efforts are frequently long-term efforts and not overnight - major changes are shocking to a team and (even if the methodology might be better suited on paper) can cause a decrease in productivity.
There's a lot more than simply effeciency or effectiveness of the process, and you can't simply look at a snapshot of the same team working in a plan-driven environment and an agile environment. You need to consider the industrial and organizational context, the attributes of the project, the team, the customer, and so on when making a decision.
Based on what I read, I'm going to have to disagree with your coworkers assessment. I'm sure that you can find some case study somewhere where an agile project was 60% less efficient with regards to some performance metric than a similar plan-driven project. However, there are also studies that show that agile yields 80% less effort, 50% less time, and high customer satisfaction with the product.
I don't have a study but I'd like to communicate my experience.
The effectiveness of any of the mentioned methodologies highly depends on the analysts.
When you've got a great product owner, then SCRUM for example is certainly faster than a waterfall approach with a bad spec.
Agile with a bad product owner is certainly slower than waterfall with a great specification.
However, more often than not, we don't know the exact requirements early enough and agile methodologies have faster feedback cycles. This means, that in uncertain terrain agile is a better method to deliver a high quality product within reasonable costs. There're numerous other advantages, for example, agile projects are easier to cancel when they don't work out and thus can reduce loss to a minimum.
One could say that agile methodologies reduce risk, while waterfall, even if it may be faster sometimes, can be quite a monetary gamble.
agile was only 60% as efficient in development time
True.
But, that's a lame measurement.
Agile methods usually deliver real value sooner.
Waterfall simply sticks to a schedule irrespective of what's delivered and often delivers nothing of value until a huge span of time has been passed.
Further.
You can measure "development time" separate from "development and test time".
Agile usually includes testing. So it seems slower.
Waterfall development can be cleanly separated from testing. So code is "ready to test" sooner. But isn't "done" until much later.
So. They're totally right. For what they measured.
I refuse to prove that Agile is more efficient,says God,for proof denies faith, and without faith Agile is nothing.