Has there really not been one thing in the past 20 years that provided huge software development gains? [closed]

Question

In No Silver Bullet, Fred Brooks makes a variety of predictions about the future of software engineering, best summed up by:

There is no single development, in either technology or in management technique, that by itself promises even one order-of-magnitude improvement in productivity, in reliability, in simplicity.

His argument is very convincing. Brooks was writing in 1986: was he right? Do developers in 2014 produce software at a rate less than 10x faster than their counterparts in 1986? By some appropriate metric -- how large has the gain in productivity actually been?

Answer 1

Do developers in 2014 produce software at a rate less than 10x faster than their counterparts in 1986?

I would imagine that there's been at least an order of magnitude improvement in productivity since then. But not by leveraging one single development, in either technology or in management technique.

Increases in productivity have come about by a combination of factors. Note: This is not a comprehensive list:

1. Better compilers
2. Vastly more powerful computers
3. Intellisense
4. Object orientation
5. Functional orientation
6. Better memory management techniques
7. Bounds checking
8. Static code analysis
9. Strong typing
10. Unit Testing
11. Better programming language design
12. Code generation
13. Source code control systems
14. Code reuse

And so on. All of these techniques combine to produce productivity gains; there isn't a single silver bullet that has ever produced an order of magnitude speedup.

Note that some of these techniques have existed since the sixties, but have only observed widespread recognition and adoption recently.

Answer 2

Robert Harvey's answer is good, but I think he left out what may be the biggest reason why programmers are more productive than ever: widespread availability of software libraries.

When I started my career there was no STL, .NET, QT, etc. You had raw C or C++, and that was about it.

Things that used to take days or weeks or work (XML parsing, TCP/IP connections, HTTP communciation) can now be done with a handful of lines of C# code. This is where we have gotten orders of magnitude better in terms of overall developer productivity.

Our current product uses a docking window framework that we purchased from a vendor. This let me have a fully functional docking window UI in a matter of minutes. I shudder to think of what it would take to do that in the days of using the win32 API.

Answer 3

For the sake of argument, I disagree with the assertion of Fred Brooks.

There is an improvement in technology which allowed alone an order-of-magnitude improvement in productivity: internet, and more precisely the progressive availability of internet connection in every home in the last two decades.

Being able to find instantly an answer to nearly every problem you can encounter as a developer enables a huge boost in productivity. Don't know how to select nth element in JQuery? Google search leads to an answer of the exact question on Stack Overflow. Don't know how to use overflow in CSS? Mozilla's MDN is here. Forgot what virtual keyword means in C#? Google helps, again.

When I started programming, I didn't have internet. I had books, as well as some digital-format documentation stored locally, but searching through books and documentation was a slow process, and no matter how much books I had, there were many issues that weren't mentioned there.

More importantly, nearly every problem you encounter was already encountered by somebody else before. Internet helps finding people who had this error and successfully solved it. This is not a sort of information you find in books or official documentation like MSDN. Instead, such information is usually found:

• On Stack Overflow, obviously. For example, I haven't seen any book which mentioned this problem.

• In blogs. I found a lot of help on blogs when I had particular problems, would it be with WCF configuration or a proxy server which doesn't start or a cryptic bug when using a specific API on a machine with culture different from en-US.

• In bug reports concerning open source software. For example, it was very helpful recently when I attempted to use Ubuntu's MAAS and when I used PXE. You don't find this sort of information in books either.

The importance of the immediate availability of an answer to most issues one can encounter is especially noticeable if we take in account that most of the time a team spends on a project is spent maintaining it.

• Internet doesn't help much during architecture and design phases (Programmers.SE might help, but it would be much more valuable for an architect to read books about architecture and design, to learn design patterns, etc.).

• It starts to be helpful during testing and implementation steps, when actual issues arise.

• But most issues appearing during the maintenance, it's there when help from others through internet appears crucial. Basic example: a WCF service works perfectly well on my machine, but fails with no exception whatsoever when deployed in production, while it worked for the last two weeks. This happened to me, and I'm thankful to blog writers and Stack Overflow community for helping me solving the issue in a matter of hours rather than weeks.

Would it justify a x10 increase in productivity? Difficult to tell.

• On one hand, there are cases where you find an answer immediately, while you could have spent days searching for it alone (or be forced to rewrite a large part of the application).

• On the other hand, the overall productivity improved based on multiple factors, such as better project management (Agile, TDD etc. comes to mind), better team management (Radical Management by Stephen Denning comes to mind), better tools (debuggers, profilers, IDEs, etc.), better hardware (no, I won't be very productive if forced to write in Assembler for a slow CPU and very limited RAM), etc.

Answer 4

I'd say the internet is a pretty good candidate. StackOverflow and Google are a modern-day developer's most powerful tools. Instant knowledge-sharing on a global basis! These days you don't need to know the answers, you only need to know how to get to know the answers - and that is a perfectly adequate subsitute that allows developers to spend less time reading and more time coding, and thus being productive. It means that every programmer in the world has access to all of the answers, and all they need to do is know how to ask the right questions.

Answer 5

I would suggest that trends pulling us in the other direction (i.e. toward lower productivity) are at least as strong as the trends you asked about. The experience of doing client/server development tool like VB6 or PowerBuilder came pretty close to the "Rapid Application Development" (RAD) ideal. You got to draw your forms, and there were obvious hooks for your procedural or SQL code.

Web development, at least initially, destroyed a lot of the techniques and infrastructure that made these things possible, and many client/server developers just stopped being developers, or clung desperately to, say, VB6.

The transition to heavily client-driven Web development was yet another slash-and-burn experience; Microsoft was getting back to the RAD ideal with WebForms, and then it quickly fell out of favor. Instead developers were expected to abuse the infrastructure (e.g. XMLHttpRequest, which is seldom used for XML) and otherwise monkey around at a low level of abstraction in an effort to make their websites more interactive

There are good reasons behind all of these transitions, and it's not fair to compare a process that yielded a native .EXE (requiring installtion and management at the individual client) to Web development, nor is it completely fair to compare a process that relies heavily on postbacks with one that yields a more seamless experience. But I will say that the practices currently in vogue result in some surprisingly low-level thought processes among people who missed out on client/server, RAD, and the like. Client/server buttons just worked, and one certainly did not have to worry about the underlying data channels that got data to the event handlers that made this happen.

Conversely, more contemporary developers tend to think that those of us who did client/server development (or even postback-based Web development) have a tendency to resort to shortcuts (and mean that in a bad way). That's understandable, but I still think that the way contemporary development is done is surprisingly low level, and the days of searching for a "silver bullet" seem to have given way to the era of mocking those of us who question the wisdom of mining and smelting our own lead.

Answer 6

Technology has advanced very much and now we have all the things Robert Harvey enumerates in his answer, but:

• The problem seems to be changing requirements. The client knows that there will not be waste of materials when changing the requirements of a software project, so they do. That kind of requirement changes almost never happen once a physical-world project like a building, is half done.

• Another important aspect is that programming continues to be highly handiwork. Rarely if ever, RAD-generated code goes into production without being modified by hand first.

• Code continues to be highly complex, and that complexity doesn't seem to decrease with new technologies.

• The rate of deadlines not met or budgets exceeded continues to be greater that those in other disciplines, and often times technical debt is incurred in order to meet them, generating hidden costs.

• Something that have, without a doubt, happened is that compartmentalization has ocurred. The sheer ammount of technologies one has to know is so that programmers have had to specialized a small number of technologies in order to become really good at them, requiring different kinds of experts to complete a large project.

• One thing that talks about software complexity is that whereas there are literally hundreds of car makers in the world, the list of companies capable of creating and maintaning an operating system, (desktop, mobile, embedded or otherwise), can be counted with the fingers of your hands.

• All the above has created a situation in which there are not enough people studying to be programmers, so that governments have created campaigns in order to motivate more students into taking that career path.

• One taste of the maturity of the software industry is that software licenses continue to state "<companyX> makes no representations about the suitability of this software for any particular purpose. It is provided "as is" without express or implied warranty." Imagine a hardware maker stating that their product is not suitable for any particular purpose. That's the state of the art right now.

Answer 7

While one might argue with specific metrics (ie, have things improved by a factor of 9.98?), I (speaking as something of an old-timer) have to agree with the general sentiment of Brooks' comment.

First off, there has been very little truly new technology invented since maybe 1970. Yes, integrated circuits have gotten longer, lower, wider, and glass fiber has improved communications speeds, but for every step forward there's one back.

Compiler technology has permitted about a 10x improvement in programmer "productivity" vs 1970, when one figures function produced divided by actual coding time, but the proliferation of new or "revised" programming languages and environments means that the average programmer is spending more and more time in "catch up" mode, and less in productive activity. Apple, Google, and Microsoft all spew out new and substantially incompatible "upgrades" to their environments at a rate that is just below that that would provoke revolt among their serfs ... er, programming customers. Similarly, HTML/CSS/Javascript/whatever keeps getting more complex.

At one time the rate at which documentation could be produced and propagated would have limited and corralled all this "innovation", but, thanks to the Internet, rigorous documentation is no longer really necessary -- just spew the functions out and rely on bloggers to ferret out the details and make them available.

Added: I've been thinking about this since yesterday, and in particular thinking about the project I worked on from about 1978 until 2008. This project (the IBM System/38 and its successors) was somewhat unique in that from the start efforts were made to control the complexity of it (one being the division of the software into two roughly equal parts, with a "machine" interface between them). In the particular area where I worked, several of my coworkers similarly were dedicated to controlling complexity (though we didn't use that term much at the time). The result was a product that (at first) was quite robust and a "hit" with the customers pretty much from the git-go. And it was a pleasure to work on -- like playing in a well-trained orchestra.

Of course, over the years complexity crept in, usually at the behest of market planners and managers who had no appreciation for controlling complexity (which is somehow different from just maintaining simplicity). I don't have the feeling that this was inevitable, but it was impossible to prevent in this case without a manager (like Glenn Henry did originally) pushing back on the forces of confusion.

Answer 8

A great deal of what we have learned in software engineering practice in the past 30+ years is of the form "technology X can speed up initial development of new software, but if you don't spend as much or more time thinking about how and when to use it as you saved by using it, in the long run it will turn your application into a sucking swamp of technical debt, costing you orders of magnitude more time and effort in maintenance."

Technologies that fall under this razor include: hand-coded assembly language, compilers, interpreters, procedure libraries, imperative programming, functional programming, object-oriented programming, manual memory allocation, garbage collection, static types, dynamic types, lexical scope, dynamic scope, "safe" pointers, "unsafe" pointers, the absence of pointers as a language concept, binary file formats, structured-markup file formats, macros, templates, avoidance of macros and templates, shared memory, message passing, threads, coroutines, asynchronous event loops, centralized remote services, distributed services, locally installed software, arrays, linked lists, hash tables, and trees.

The fact that many of the items in the above list come in groups which together exhaust the known solution space is very much intentional, and should in and of itself tell you something. One could argue that the only unambiguous, across-the-board improvements in praxis we've had since they first switched on the Z3 are block-structured programming (as opposed to spaghetti code) and memory protection (boy, do I ever not miss the days when a typo could take down my entire computer).