The C2 wiki has a discussion of Empirical Evidence for Object-Oriented Programming that basically concludes there is none beyond appeal to authority. This was last edited in 2008. Discussion here seems to bear this out: questions on whether OO is outdated, when functional programming is a bad choice and the advantages and disadvantages of AOP are all answered with contributors' opinions without reliance on evidence.

Of course, opinions of established and reputed practitioners are welcome and valuable things to have, but they're more plausible when they're consistent with experimental data. Does this evidence exist? I know that evidence-based software engineering is a thing, but can I practice it in this context? Specifically, if I have a particular problem P that I want to solve by writing software, does there exist a body of knowledge, studies and research that would let me see how the outcome of solving problems like P has depended on the choice of programming paradigm?

I know that which paradigm comes out as "the right answer" can depend on what metrics a particular study pays attention to, on what conditions the study holds constant or varies, and doubtless on other factors too. That doesn't affect my desire to find this information and critically appraise it.

It becomes clear that some people think I'm looking for a "turn the crank" solution - some sausage machine into which I put information about my problem and out of which comes a word like "functional" or "structured". This is not my intention. What I'm looking for is research into how - with a lot of caveats and assumptions that I'm not going into here but good literature on the matter would - certain properties of software vary depending on the problem and the choice of paradigm.

In other words: some people say "OO gives better flexibility" or "functional programs have fewer bugs" - (part of) what I'm asking for is the evidence of this. The rest is asking for evidence against this, or the assumptions under which these statements are true, or evidence showing that these considerations aren't important. There are plenty of opinions on why one paradigm is better than another; is there anything objective behind any of these?

  • 1
    web search for evidence-based software engineering shows plenty links
    – gnat
    Commented Aug 29, 2012 at 13:07
  • @gnat EBSE is about systematically summarising the literature and drawing general conclusions about how we could improve the practice. My question is whether that literature exists; whether there's enough for systematic reviews or metaanalyses to be productive.
    – user4051
    Commented Aug 29, 2012 at 18:45

7 Answers 7


For the previous take, see Revision 1 of this answer. However, the comments and edits to the question further clarify what the question is seeking and allow me to be more clear.

Yes, evidence based software engineering (EBSE) is a thing. There appears to be a few efforts toward EBSE databases, such as this one at Durham University and SEED, which was started by a professor at Cal Poly. All of these efforts, as well as ones discussed in a number of papers that can be found through the IEEE Xplore server or the ACM Digital Library (subscription or purchase required for many papers in both), are based on evidence-based medicine. They provide literature reviews of published empirical (observation and experiment) data. In fact, including "literature review" in a search string on any publication search yields information on most topics - over 14000 hits on the ACM and over 1000 on the IEEE database (when limited to only computing topics).

Looking at the general types of topics discussed in these EBSE databases and literature reviews, I see a common thread - they tend to be technology-independent. The emphasis appears to be mostly centered around process and methodology rather than the specific tools used to conduct software engineering.

So, this concept exists in software engineering. Academia is aware of the evidence-based concept and can successfully apply it to software engineering.

Specifically, the question addresses applying EBSE techniques to the selection of a paradigm seems difficult, due to the sheer number of variables involved, forcing assumptions to be made as well as reducing the ability to repeat the experiment or observation. It's said right in the question - "which paradigm comes out as "the right answer" can depend on what metrics a particular study pays attention to, on what conditions the study holds constant or varies, and doubtless on other factors too". Although that doesn't mean that studying which paradigm is "best" in a given situation, it makes any kind of literature review of these documents incredibly difficult to complete and still extract information across them.

There definitely isn't a "turn the crank" solution to choosing a paradigm.

Given a programming paradigm, you can find studies in the various academic and industry databases about how that paradigm influences various aspects of software development - quality, defects, efficiency, and so on - under various specific conditions, ranging from the knowledge and experience of the team to the problem domain. Any rigorous paper should clearly identify the conditions under which the data was gathered and the assumptions. The problem becomes trying to isolate the factors that make it good in each of those conditions.

Academically, there are some statements that are easy to research. For example, the statement that the functional paradigm is well suited to applications that require concurrency stems from the Church-Rosser theorem. Because of this, it's likely that a software system written in a functional language will have fewer defects related to concurrency than the same system written in a procedural or object-oriented language.

However, from a practical standpoint, a software team can't always use "the best" tool or technique for the job just because research indicates it. Although we strive for producing the highest quality software systems, we do operate within constraints. Often, I see these constraints minimized (or even removed from the equation) when discussing the effectiveness of any methodology.

As a practitioner, when I'm involved in choosing technologies to use, I try to identify the best possible tools. But then I constrain myself to what is known and well-understood by the team that I have. Going back to my previous example, if I have a team well-versed in building concurrent applications in C++ and no experience in Haskell, it doesn't make sense to propose building the system in Haskell as I likely won't be able to make schedule and budget constraints, and my quality will likely suffer due to a lack of experience in the toolchain.

To recap, evidence-based software engineering is generally a good thing that exists and literature reviews do exist and are readily available. However, there are aspects of software engineering where applying evidence-based approaches offer little value. The selection of a programming paradigm to a large scale development effort is one of these.

If you want to find out about how object-orientation addresses reusability or defects in functional programming - you'll easily find publications on those. However, I didn't find (nor would I put any amount of trust in) a publication that was able to effectively address paradigm selection across the broad range of real-world software engineering projects.

  • The section on Turing completeness ignores trade-offs, which are what I want to see brought out into the open and compared. Let me give a specific example. Plenty of people tell me that functional programming is great for avoiding concurrency bugs. Now we find that Scheme is, like Pascal, Turing-complete. So it should be possible to write concurrency-safe code procedurally. Agreed. But is it great? Are there advantages to choosing one method? Can such advantages be measured?
    – user4051
    Commented Aug 29, 2012 at 16:41
  • 1
    @GrahamLee Confirming or rejecting your hypothesis requires objective evidence, which does not exist. There are subjective reasons to create a new paradigm and a new model of representing the exact same computational capabilities - and this is not restricted to programming languages, but also to the underlying mathematical representation of said languages. Those objective reasons include targeting a particular demographic (computational mathematician versus business analyst - their way of thinking is different requires a different paradigm).
    – Thomas Owens
    Commented Aug 29, 2012 at 17:48
  • 2
    @Thomas: your implication that programming practices are uniquely opaque to science is peculiar. There's plenty of research going on. An oft-quoted example is Lutz Prechelt's research group. I don't know the area well enough to know if anyone's tackled Graham's specific questions, but there's no reason to believe that they are not tractable to the sort of methods used by Prechelt and others. Commented Aug 29, 2012 at 20:18
  • 1
    @Chris I don't believe that they are opaque to science. However, I believe that there are some things that by the time you, as Graham says in the question, add "a lot of caveats and assumptions" to the research, it's no longer useful to practitioners. At that point, from a practical, in-the-trenches perspective, it's simply more effective to base your decisions on history and experience. Having good, hard, valid data is a very good thing. But there comes a point when it's too hard to get or it's only valid under a very specific situation that it isn't useful to industry.
    – Thomas Owens
    Commented Aug 29, 2012 at 20:23
  • 1
    @Thomas I doubt that. Medical general practice is at least as situational and context-sensitive as is software engineering, and, er, evidence is mounting that evidence-based GP produces measurable improvements. It's largely a matter of the quantity and quality of research. Commented Aug 29, 2012 at 20:33

I've been reading The Art of Unix Programming by Eric S. Raymond. It has some very interesting historical insights about things we now take for granted. He cites some good studies from IEEE software that use empirical evidence like defect density. That might be a good source if you're looking for academic-style studies.

Even techniques like modularizing using functions were not always common practice. One of my favorite quotes from the book so far:

Dennis Ritchie encouraged modularity by telling all and sundry that function calls were really, really cheap in C. Everybody started writing small functions and modularizing. Years later we found out that function calls were still expensive on the PDP-11, and VAX code was often spending 50% of its time in the CALLS instruction. Dennis had lied to us! But it was too late; we were all hooked...

-- Steve Johnson

There are really two problems with getting too empirical, though. The first is that code quality is a very subjective thing. Code can be terrible and still be correct. People's perception of a programming paradigm is a very valid metric, because code is written for people to read as much as for computers, if not more.

The second problem is that 50% of developers have below average programming talent. It doesn't matter if your top developer is more productive using functional programming if the "rabble" struggles to write working software using it, let alone beautiful, well-architected software. Likewise with TMTOWTDI programming languages, your top developer is still going to write clean, modular code, but less talented coders write line noise due to the lack of imposed structure.

That's why I think OOP has risen to the top in popularity despite its deficiencies. It's not so restrictive that it hobbles the most talented, but its structure provides a concise way to communicate and impose good design principles on the least talented.

In our line of work we have a tendency to evaluate a solution based too much on its technical merits alone. A successful endeavor has to account for the human side of the equation as well.

  • "code quality is a very subjective thing" agreed - you have to be careful what you measure, and perception is an important factor. But perception, like many other things, is malleable too: look at the rise and fall and rise of functional programming to see that what people think of how they work is not directly related to how they work. I've also recently been re-reading TAOUP. Part of my motivation for this question is seeing in early literature solutions to problems that are current in software engineering.
    – user4051
    Commented Aug 29, 2012 at 16:38
  • +1, "The second problem is that 50% of developers have below average programming talent." This sentence is a relief to me. It is better than many pills I tried :)
    – NoChance
    Commented Aug 29, 2012 at 17:29

There are programming contests that use a computer grading system and allow you to write in various languages and post all kinds of results and things. I bet they have good data for you. Here's a list of 8: http://www.makeuseof.com/tag/8-onlineprogramming-contests-challenge-win/

I imagine you can make meaningful comparisons of solutions to very simple and clear-cut problems, like sum of squares, or Fibonacci series, or drawing a straight line using the Bresenham's line algorithm. Most real-world programming tasks do not have such clear-cut goal posts and every language has its sweet spots. Much of the benefit of a language is subjective. You might find more meaningful data by surveying programmer and client happiness than by counting lines of code or numbers of defects.

I remember when I spent a half day writing one of my first Awk programs, I thought it would have taken me a whole week to do the "same" thing in Java. But that's because my Java solution would have focused on being robust where as the Awk solution was quick and dirty and required some manual tweaking on the input and output and was really throw-away when I was done. Awk and Java are both great, just not for the same things.

I guess what I'm saying is that for real world applications, comparing languages or tools in a meaningful way is extremely difficult - the old apples and oranges issue. Good luck! I'd love to see what you find out.


I've been studying different ways to develop software for 30+ years. There is a dearth of good published evidence on choosing a paradigm.

I put together a large searchable ASCII bibliography. This includes a lot of IEEE and ACM papers and articles. I tag the items with the type of evidence provided. Here are the most common tags:

233 =IDEA
338 =ESSAY
500 =DEMO

Now search for PARADIGM and count the tags

  1 =ESSAY
  1 =IDEA
  1 =TALK

If you want to dig deeper, http://cse.csusb.edu/dick/lab.html and I hope it helps...


It seems that in many cases there isn't a corpus of research big enough or of high-enough quality to allow general conclusions to be drawn about whether one practice in software engineering is better than another. I was specifically looking for research into working in different paradigms, but the lack of availability is not limited to that arena so I'll frame my answer in a broader sense.

A paper from 2004, Evidence-Based Software Engineering by Kitchenham et al, covers quite succinctly the benefits to be derived from an evidence-based approach and the problems with implementing it in software engineering. I won't discuss the benefits side here (it's clear from the question that I would like to be able to work in this way), but the problems are relevant as an answer to the question I asked.

  • firstly, if you're not a member of the ACM then you probably can't read the link above at all, which covers the first problem: not all of the extant research is actually available to practitioners.
  • much software engineering practice goes on in secret as part of commercially-confidential processes so there's no visibility onto what did or didn't work for those people.
  • software engineering is a skilled practice, so it's hard (not impossible, merely hard) to arrange a suitably-blinded study.
  • the different parts of the software lifecycle affect each others' outcomes to an extent that's hard to control for in any experiment.
  • as evinced by the discussions here, many practitioners do not see "the literature" (or the academic side of software engineering in general) as relevant to their work.

So the answer I'm after is "no", the evidence I'm looking for probably doesn't exist. I should choose my paradigm based on the existing popular criteria of what I know, what's cool and expert opinion.

  • 2
    from the abstract of the cited paper, emphasis mine: "The skill factor means software engineering experiments are vulnerable to subject and experimenter bias. The lifecycle factor means it is difficult to determine how technologies will behave once deployed. Conclusions: Software engineering would benefit from adopting what it can of the evidence approach provided that it deals with the specific problems that arise from the nature of software engineering." To which I would add: good luck with that! ;) Commented Aug 31, 2012 at 19:49
  • Steven: part of the motivation behind EBSE is to go from "I can guess the following problems, therefore I will reject any chance of your solution working" to analysing results on their own merits. There's much more to a paper than its abstract.
    – user4051
    Commented Sep 7, 2012 at 18:43
  • 2
    I appreciate your enthusiasm. Medical science and software development are radically different disciplines. While the analogy is interesting, it is hardly groundbreaking. The full paper is available here labada.inf.utfsm.cl/~gvaldes/ESE/docs/… Section 5 reflects the impedance mismatch mentioned in the abstract. A more direct mapping of the medical techniques would be to debugging existing systems, not building new ones. ;) If you want to build better products, build better teams. The people are far more important than the tools (c.f. Peopleware) Commented Sep 9, 2012 at 4:33
  • 1
    addendum: the EBSE site has some useful information, dur.ac.uk/ebse/evidence.php would be especially useful to those new to the SE field - but take the surveys with a block of salt, because (1) the evidence available is scant and (2) the average results may not be relevant to the performance of your team of specific individuals with highly specialized skills and aptitudes. Commented Sep 9, 2012 at 4:40

I do not believe this type of study exists. One would believe it is not the programming paradigm that matters as much as the actual algorithm that is used. For instance given any non-trivial system one that relies on small space algorithms verse one that relies on small time algorithms would generate different metrics. The one that has the better time would most likely be deemed more valid, unless space is an issue then the inverse is true. I find it similar to paving a road. While the algorithm or recipe for making the materials is constant throughout all processes it would be possible one company thinks paving two lanes at once (one on each side of the road) is better than paving two lanes on the same side of the road at once. At the end of the day it doesn't matter as the process of making the black top is still the same, the only difference is the approach. Going back to programming, if you have a team of C developers, write the code in a procedural manner, if you have a team of Java developers write it in OO. Do not get hung up on the paradigm so much as the implementation of the algorithm. Because at the end of the day you can write Java like C and you can try to write C like Java.


To reply to the comment graham left me:
I assume by architecture you mean programming paradigm. If you are going to use Clojure maybe you should hire a team of Clojure programmers. However, based on a quick search Clojure is a Java based language it just so happens to be functional. Given that information I would take the Java programmers (since technically they can just write Java and it will give you the same results) or look for functional programmers such as Haskell developers. Now in terms of choosing what is best it is completely dependent on your team. I would never have a team of relational database experts organize a cloud solution for me nor would I have a team of functional programmers build out an object orient solution for me. You have to use the strengths of the team you have not the glorified vision you have in your head for what a team "should" be able to do.

  • How do I choose whether to hire a team of Java programmers, or a team of C programmers? Should I retrain them in Clojure? Once I've chosen my algorithm, what architecture is the "best" way to encapsulate it in a software solution, for given values of "best"?
    – user4051
    Commented Aug 29, 2012 at 13:08
  • @GrahamLee see my update
    – Woot4Moo
    Commented Aug 29, 2012 at 13:15
  • I feel we're discussing the same problem but at different levels of abstraction. "Using the strengths of the team you have" would've meant never building a computer, because Bletchley Park didn't have anybody with strength in building computers. Sometimes you have to say "we could create a better solution if we used this thing"; what I'm after is the information about those cases.
    – user4051
    Commented Aug 29, 2012 at 13:28

Different paradigms lead to different solutions. Which fit is 'best' depends largely upon:

  • the solution
  • the development team
  • the operational environment

I know of no such definitive study, and even if there was one I would not trust it.

That is the architect's job.

Replacing the architect's wisdom with the possibly irrelevant conclusions of a study is a recipe for disaster.

Note: a comment mentions deciding upon "the algorithm" and then choosing the language. Algorithms are the central structural mechanism for procedural languages. Object-oriented languages focus on classes and patterns of collaboration/communication. If you're convinced (as the architect) that an algorithm-centric solution is 'best', then stick with procedural or functional languages.

Addendum: not trusting studies is not 'superstition', it is common sense. Scientific experiments must be objective and repeatable. Software projects are highly subjective, but even worse they are unrepeatable experiments. You simply cannot implement a project X with team Y, measure the results, and then roll back time, erase memories, and re-do the same project with the same team. Information discovered or implied by studies may be useful to the architect, but it can never be definitive.

  • 1
    So is it outside the purview of an architect to look for prior work on which to build his or her opinion? Probably not - hence the question of whether and where such results can be found.
    – user4051
    Commented Aug 29, 2012 at 13:15
  • 2
    If there was a definitive study with a reasonable experimental method the results of the study might be interesting; as it stands this answers seems to state that any study is worthless irregardless of method which is a little too non-scientific for my liking
    – jk.
    Commented Aug 29, 2012 at 13:24
  • 3
    @Steven: the word for not trusting the results of a truly definitive study is 'superstition'. Perhaps what you really mean is you don't believe there can ever be definitive studies in SE (which statement would itself obviously require a large, well-supported body of evidence). Commented Aug 29, 2012 at 21:27
  • 1
    The quality of a software method is in how well it matches the local human need. In general, it is not constrained by the laws of physics (Scotty). It will be a long while before 'software' [discipline] manages to distil its immutable fundamental laws. For example see "Software Quality Metrics: Three Harmful Metrics and Two Helpful Metrics" in ppi-int.com/newsletter/SyEN-046.php#feature and ppi-int.com/newsletter/SyEN-047.php#feature Commented Aug 31, 2012 at 17:22
  • 1
    @Cris: For the record, no, I do not believe that there can ever be a truly definitive study in this area; see addendum. The idea that a 'definitive' study can be used instead of expert judgement in order to make a critical architectural decision is 'appeal to authority', which is a form of logical fallacy :) In my experience - and I'm not making blanket accusations, just an observation - the quest for such things is most often either an attempt to avoid responsibility for a decision, or an attempt to support a decision that has already been made. Commented Aug 31, 2012 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.