6

Everyone seems to implicitly assume that the free market of ideas will eventually converge on the "right" solutions in software development. We don't assume that in medicine - we recognise that scientific experiments are needed there - so why should we assume it in software development?

I am not arguing for regulation of programmers. It is far too early to even talk about that. Before healthcare could be effectively regulated, there was a need for scientific experiments to establish which treatments worked and which didn't.

Software engineering doesn't even have this scientific evidence base to back up touted methodologies such as Scrum or Agile, or programming paradigms like functional programming or MDA. As

(a) large software projects are responsible for many government project failures (with the UK government being a really good example)

(b) Agile and Lean are being used outside of software development, including in the public sector [of course, Lean originated outside of software development]

this is increasingly politically relevant. Government project failures may be influenced by a failure to use a best practice, or even by using something that is considered by some people to be a best practice, but which actually makes things worse, or just costs money without really helping very much.

The question is, why is this scientific evidence base (for all intents and purposes) nonexistent?

There is a large open source community from which research participants could be drawn. My fear is that the closed-source and in-house software developers would treat with suspicion any research based on this community, fearing (perhaps rightly) that the results would not translate over. And companies that develop closed-source and in-house software would probably not be willing for their developers to participate in any scientific studies. For one thing, it would probably take time away from getting work done; for another, the results could be embarrassing to the company or to senior managers.

Improve this question
4
  • 3
    possible duplicate of Are there any studies on the Efficiency/Effectiveness of Agile vs Waterfall
    – Tom Squires
    Mar 17, 2012 at 11:04
  • It looks like people are not getting the message. One of the answers to that question cited "case studies" (aka anecdotes). Another answer flat out stated "I don't know of any studies, but here is my personal experience". This may all give you warm fuzzies, but it is not scientific.
    – Robin Green
    Mar 17, 2012 at 13:33
  • @TomSquires I think it's similar and related, but not identical. That question that you linked to is explicitly about methodologies. This question is about empirical data in software engineering in general.
    – Thomas Owens ♦
    Mar 17, 2012 at 14:07
  • Wow, some of the answers on here are frightening in terms of a post modern rejection of the need for scientific evidence because "everybody knows" agile is better than waterfall. Firstly, it isn't just a case of one methodology over another we see this attempt to reframe agile being good because waterfall as bad when I see waterfall more of an attempt to control budgets when delivering projects versus agile as being more about overly engaging users - users have day jobs too. My background is scientific moving into IT and have never been able to get my head around most internal IT's use of agil
    – Zak Willis
    Feb 5, 2018 at 9:07

3 Answers 3

Reset to default
5

There are decent libraries of research into software development projects. Just take a look at the IEEE Computer Science Digital Library and the ACM Digital Library for two examples. More specifically, the IEEE Transactions on Software Engineering and ACM Transactions on Software Engineering and Methodology present academic and industrial research into various aspects of software development, ranging from tools through methodologies and paradigms. In addition, IEEE Software is more of a magazine format that presents relevant topics and information in a easier to digest format for professionals "in the trenches", so to speak. So there is research out there.

However, one of the biggest problems is variation in the software industry. Projects depend on so much - domain, knowledge and skills of the people involved, process methodology, the choices made at various phases of the project (from the technology to use to the system architecture and design - there are often many good choices). It becomes very difficult to look at projects and generalize information in such a way that it is both scientifically valid and useful across the majority of projects.

Your comments about cost and embarrassment are also probably true, at least in some organizations. Organizations pay their engineers to product shippable products. From what I've seen, the majority of the research involving commercial products involves three things: a description of the technique before some change, a description of some technique after a change, and the impact of this change. Going back to embarrassment, some companies aren't going to publish reports that say their productivity decreased or their defects increased and analyze why this happened. Instead, they are going to tend to show off techniques that improved the organization and discuss why. However, in science, the failures are just as important as the successes, especially when something that didn't work on one project can be improved or adopted and work extremely well on another project.

Improve this answer
4
  • 1
    People are very different too, and yet randomised controlled trials can be used to account for differences between people in medical and psychological studies. In principle, differences between teams should not be an insurmountable obstacle.
    – Robin Green
    Mar 17, 2012 at 13:27
  • 1
    @RobinGreen That's true, and some studies do take a psychological or sociological approach. However, there are many variables in a project, ranging from people to tools. By the time you start reducing the variability, you end up with data that's interesting academically, but all you did was describe one set of projects with one group of people using one set of tools and processes. It becomes harder for me to determine if it's applicable to my people using my tools and processes. And to compound it, companies might be adverse to experimenting with projects that they are relying on to make money
    – Thomas Owens ♦
    Mar 17, 2012 at 14:05
  • @RobinGreen: It is (literally) exponentially more difficult to track the development of something as it depends on many people who can all individually be different, compared to a patient (who is just one person who can be different). If you assume people come in 10 varieties (as a very oversimplified example), a patient study accounts for 10 different kinds of people, and a development study would have to account for 10^amount_of_team_members. With more possible variations comes the (exponentially increased) need for more data to base your observations off.
    – Flater
    Jul 20, 2022 at 9:01
  • 1
    “ However, in science, the failures are just as important as the successes” - it’s sad, but this is absolutely not true, especially in the PhD field. It is very common to steer away from results that do not match your hypothesis and emphasize results that do - if you don’t have any interesting results you are decreasing your chance of being published and you can’t get your PhD. A big flaw.
    – Dirk Boer
    Jul 20, 2022 at 20:07
1

In general, I'd say that it's pretty much impossible to draw any meaningful real-world scientific conclusions because every (non-trivial) software project is unique and software development is, by its very nature, non-deterministic. So I don't think scientific studies would give much useful information.

Improve this answer
2
  • That sounds plausible at first glance. However, suppose the hypothesis is that "Scrum, correctly applied, results in fewer cancelled projects than Waterfall". Are you really saying that it's impossible to get useful reliable data on this question, and we have to go with our gut instincts?
    – Robin Green
    Mar 18, 2012 at 7:17
  • @Robin You don't need any studies to show that agile (iterative, incremental, collaboration) is vastly superior to waterfall (phased, sequential). That already established and not something we need to dwell on.
    – Martin Wickman
    Mar 21, 2012 at 11:03
1

Wouldn't that be great if there were. This would probably come from the universities, but they're battling the problem of not producing programmers who know what they're doing when they graduate. Do we see them as teaching theory and doing research or doing technical job training. If you can't demonstrate graduates get jobs, enrollment/money drops. The money for research has to come from somewhere. You'd think the less technical business managers would be interested in funding the research, so they could use it to base decisions. Applying scientific research in business is just scratching the surface.

The medical profession is a good analogy. When someone graduates from medical school (theory and research), they're not ready to be practicing doctors. They do internships and a residency for several years and possibly go into specializations. This happens at the hospitals. Companies that hire jr. devs need to understand they are still "practicing" programming and require mentorship and training. This could free up the CS departments to devote more time to researching the latest and greatest.

A lot of programmers are in jobs where they don't get to consider the hyped technologies whether there's any data to support them or not. Unlike switching a patient's prescription, rebuilding legacy applications may not be cost effective. How much time does a doctor invest in learning about the new drug compared to what it would take a programmer to learn a new platform or language?

What do we expect the research to tell us? If you fully understand a technology, apply it properly in the right situation, it will work the best. We're in a specialized industry. Yes, your product is as good as you've "hyped" it, but I can't hire enough devs that know how to use it and most projects don't have enough time for learning new things.

Improve this answer
2
  • "What do we expect the research to tell us?" - This is exactly the problem with the unscientific mindset. Assuming that you already know the answer. Well maybe you don't.
    – Robin Green
    Mar 17, 2012 at 13:31
  • I think you vastly underestimate the amount of time, effort and money goes into your medical analogy. I'm pretty sure its easier for me to convince programmers to rebuild a legacy app than it is for me to convince doctors to change many of their practices.
    – Fomite
    Mar 17, 2012 at 23:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.