Is there a correlation between some software engineering practices and software engineering success stories?

Question

I've been reading the book "The Drunkard's Walk: How Randomness Rules Our Lives" by Leonard Mlodinow and it's a truly enlightening read. The book deals with probabilities and human reasoning. And let's just say, for the record, that while some things sometimes work, there's a chance that the things you thought made it work, are unrelated to what actually made it work.

Probabilities are unintuitive.

However, it gave me an idea. There ought to be studies on this already, that have attempted to quantify the results of software engineering endeavors (which of course is a hard problem in itself). And these studies should point to what kind of software engineering practices are really important in terms of quantifiable success.

i.e.

• A team that employs TDD is this much less likely to have this kind of problem.
• A team that employs SOLID principles is this much less likely to have this kind of problem.
• etc. etc.

What I'm looking for here is software engineering practices that show strong correlation between implementation and success. I'm confident that these things exist but that they are hard to come by and that's why I pose this question.

What studies or what practices do you know of that have strong correlation between implementation and success (where success is somewhat arbitrary but I think you get the idea)?

If we're gonna sell that idea that software engineering is better than cowboy coding, I think we need proofs.

Answer 1

The problem with this kind of quantification is that it's almost impossible to get good enough data on the effectiveness of software engineering practices to make any meaningful conclusion.

Most importantly, correlation does not imply causation - for example, it could just be that good programmers are quick to jump on and implement new ideas, so you see a general correlation between project success and adoption of new software engineering techniques. But that proves nothing about the effectiveness of the techniques themselves, as the entire effect might be explained by the higher talent level of the programmers who adopt them.

And then it's hard to control the independent variables. How do you ensure a fair experiement unless you are able to control for all the following?

• Experience / skill / motivation level of team
• Actual extent of adoption of claimed methodology (are they really doing TDD properly?)
• Presence / absence of major design mistakes unrelated to the software engineering methodology (e.g. those requiring a major re-architecture during the project)
• Difficulty level of projects being compared
• Impact of externally imposed problems (e.g. major requirements changes)
• Selection bias (e.g. did people tend to share data more often about successful projects?)
• Confirmation bias (e.g. did people exaggerate the success of projects which use their favourite methodology?)

Even if you decide to tackle the above by giving multiple carefully selected teams the same problem under the same carefully controlled conditions then your experiment is likely to be prohibitively expensive if you want to create enough data to be statistically significant.

And finally, it's almost impossible to measure success:

• A quantity metric like source lines of code (SLOC) is appallingly bad. The incentive on developers is to create million-line monstrosities with copy / paste coding in order to look more "productive"
• An on-time / on-budget metric depends mainly on the level of ambition in the estimates used to create the plan / budget
• A ROI-type metric depends as much on market situation and the commercial management of the product as it does on the quality of the engineering output (look at the history of Microsoft Windows!)
• Story Points are useful to get a feel of velocity in an agile team but aren't really comparable across teams
• Functionality based metrics like Function points or Use Case Points are perhaps the best of a bad bunch, but they are bureaucratic to collect and don't reflect the difference in engineering effort required to create each unit of functionality.
• Quality metrics like bugs in production / app availability are notoriously difficult to calculate and compare on an equal basis - it depends significantly on things like platform chosen, size of user base and various operational / deployment factors.

In conclusion: trying to quantify the impact of software development tasks is an extremely difficult task, and despite many years following the topic nobody has yet come up with a truly effective approach. As a result, evaluation of software development methodologies remains more of an art than a science, and probably will remain so for many years to come.

Interestingly, there is one approach that I think has promise: application of lean principles. This isn't a panacea and won't directly solve the problem of evaluating software development methodologies, but it does have one key insight: A process with a particular element of waste is unambiguously less efficient than the same process without that element of waste, all other things being equal. So if you focus on eliminating waste in the software development process, you can at least be sure you are moving in the right direction. In addition, waste is often quantifiable so you should also get some idea of how much more efficient you are getting, at least in rough percentage terms.

Answer 2

There are indeed, attempts at quantifying these with studies, however, they are often pay-walled (like the CHAOS report by Standish group) that, according to some reviewers, show Agile projects are more likely to be successful (42% vs 13%) than Waterfall ones (claimed to be measured by six individual attributes of success: OnTime, OnBudget, OnTarget, OnGoal, Value, and Satisfaction).

On the other hand, a report from the International Software Benchmarking Standards Group (ISBSG) data repository of completed software projects claims that only in large projects the Agile method seems more productive than traditional waterfall development methods (the report defines a project as "large" if it has between 1000 and 3000 IFPUG 4+ and Nesma function points). Unfortunately the ISBSG raw data is also pay-walled.

On the other hand, statements backing up the above reports are conspicuous by their absence in works like in Evidence-based Software Engineering open book