Of the controlled experiments, only three show an effect large enough
to have any practical significance. The Prechelt study comparing C,
C++, Java, Perl, Python, Rexx, and Tcl; the Endrikat study comparing
Java and Dart; and Cooley’s experiment with VHDL and Verilog.
Unfortunately, they all have issues that make it hard to draw a really
In the Prechelt study, the populations were different between dynamic
and typed languages, and the conditions for the tasks were also
different. There was a follow-up study that illustrated the issue by
inviting Lispers to come up with their own solutions to the problem,
which involved comparing folks like Darius Bacon to random undergrads.
A follow-up to the follow-up literally involves comparing code from
Peter Norvig to code from random college students.
In the Endrikat study, they specifically picked a task where they
thought static typing would make a difference, and they drew their
subjects from a population where everyone had taken classes using the
statically typed language. They don’t comment on whether or not
students had experience in the dynamically typed language, but it
seems safe to assume that most or all had less experience in the
dynamically typed language.
Cooley’s experiment was one of the few that drew people from a
non-student population, which is great. But, as with all of the other
experiments, the task was a trivial toy task. While it seems damning
that none of the VHDL (static language) participants were able to
complete the task on time, it is extremely unusual to want to finish a
hardware design in 1.5 hours anywhere outside of a school project. You
might argue that a large task can be broken down into many smaller
tasks, but a plausible counterargument is that there are fixed costs
using VHDL that can be amortized across many tasks.
As for the rest of the experiments, the main takeaway I have from them
is that, under the specific set of circumstances described in the
studies, any effect, if it exists at all, is small.
Moving on to the case studies, the two bug finding case studies make
for interesting reading, but they don’t really make a case for or
against types. One shows that transcribing Python programs to Haskell
will find a non-zero number of bugs of unknown severity that might not
be found through unit testing that’s line-coverage oriented. The pair
of Erlang papers shows that you can find some bugs that would be
difficult to find through any sort of testing, some of which are
severe, using static analysis.
As a user, I find it convenient when my compiler gives me an error
before I run separate static analysis tools, but that’s minor, perhaps
even smaller than the effect size of the controlled studies listed
I found the 0install case study (that compared various languages to
Python and eventually settled on Ocaml) to be one of the more
interesting things I ran across, but it’s the kind of subjective thing
that everyone will interpret differently, which you can see by
This fits with the impression I have (in my little corner of the
world, ACL2, Isabelle/HOL, and PVS are the most commonly used provers,
and it makes sense that people would prefer more automation when
solving problems in industry), but that’s also subjective.
And then there are the studies that mine data from existing projects.
Unfortunately, I couldn’t find anybody who did anything to determine
causation (e.g., find an appropriate instrumental variable), so they
just measure correlations. Some of the correlations are unexpected,
but there isn’t enough information to determine why.
The only data mining study that presents data that’s potentially
interesting without further exploration is Smallshire’s review of
Python bugs, but there isn’t enough information on the methodology to
figure out what his study really means, and it’s not clear why he
hinted at looking at data for other languages without presenting the
Some notable omissions from the studies are comprehensive studies
using experienced programmers, let alone studies that have large
populations of “good” or “bad” programmers, looking at anything
approaching a significant project (in places I’ve worked, a three
month project would be considered small, but that’s multiple orders of
magnitude larger than any project used in a controlled study), using
“modern” statically typed languages, using gradual/optional typing,
using modern mainstream IDEs (like VS and Eclipse), using modern
radical IDEs (like LightTable), using old school editors (like Emacs
and vim), doing maintenance on a non-trivial codebase, doing
maintenance with anything resembling a realistic environment, doing
maintenance on a codebase you’re already familiar with, etc.
If you look at the internet commentary on these studies, most of them
are passed around to justify one viewpoint or another. The Prechelt
study on dynamic vs. static, along with the follow-ups on Lisp are
perennial favorites of dynamic language advocates, and github mining
study has recently become trendy among functional programmers.