Why can't the IT industry deliver large, faultless projects quickly as in other industries?

Question

After watching National Geographic's MegaStructures series, I was surprised how fast large projects are completed. Once the preliminary work (design, specifications, etc.) is done on paper, the realization itself of huge projects takes just a few years or sometimes a few months.

For example, Airbus A380 "formally launched on Dec. 19, 2000", and "in the Early March, 2005", the aircraft was already tested. The same goes for huge oil tankers, skyscrapers, etc.

Comparing this to the delays in the software industry, I can't help wondering why most IT projects are so slow, or more precisely, why they cannot be as fast and faultless, at the same scale, given enough people?

---

Projects such as the Airbus A380 present both:

• Major unforeseen risks: while this is not the first aircraft built, it still pushes the limits of the technology and things which worked well for smaller airliners may not work for the larger one due to physical constraints; in the same way, new technologies are used which were not used before, because for example they were not available in 1969 when the Boeing 747 was certified.

• Risks related to human resources and management in general: people quitting in the middle of the project, inability to reach a person because she's on vacation, ordinary human errors, etc.

With those risks, people still achieve projects like those large airliners in a very short period of time, and despite the delivery delays, those projects are still hugely successful and of a high quality.

When it comes to software development, the projects are hardly as large and complicated as an airliner (both technically and in terms of management), and have slightly less unforeseen risks from the real world.

Still, most IT projects are slow and late, and adding more developers to the project is not a solution (going from a team of ten developers to two thousand will sometimes make it possible to deliver the project faster, sometimes not, and sometimes will only harm the project and increase the risk of not finishing it at all).

Those which are still delivered may often contain a lot of bugs, requiring consecutive service packs and regular updates (imagine "installing updates" on every Airbus A380 twice a week to patch the bugs in the original product to prevent the aircraft from crashing).

How can such differences be explained? Is it due exclusively to the fact that the software development industry is too young to be able to manage thousands of people on a single project in order to deliver large scale, nearly faultless products very fast?

Answer 1

Ed Yourdon's Death March touches upon a number of these meta type questions.

In general, the software industry lacks a lot of the following, which gets in the way of large projects.

• Standardization and work item breakdown.

  • This has certainly gotten better, but the design constructs still aren't there to break out a big system. In some ways, software can't even agree on what's needed for a given project, much less being able to break things down into components.
  • Aerospace, building construction, auto, etc.. all have very component-driven architectures with reasonably tight interfaces to allow fully parallel development. Software still allows too much bleed through in the corresponding areas.

• A large body of successful, similar projects. The A380 wasn't the first big airplane that Airbus built. There are a lot of large software applications out there, but many of them have suffered dramatically in some aspect or the other and wouldn't come close to being called "successful."

• A large body of designers and builders who have worked on a number of similar and successful projects. Related to the successful project issue, not having the human talent who has been there, done that makes things very difficult from a repeatability point of view.

• "Never" building the same thing twice. In many ways, an airplane is like any other airplane. It's got wings, engines, seats, etc.. Large software projects rarely repeat themselves. Each OS kernel is significantly different. Look at the disparity in file systems. And for that matter, how many truly unique OSs are there? The big ones become clones of a base item at some point. AIX, Solaris, HP-UX, ... herald back to AT&T System V. Windows has had an incredible amount of drag forward through each iteration. Linux variants generally all go back to the same core that Linus started. I bring it up, because the variants tend to propagate faster than the truly unique, proprietary OSs.

• Really bad project estimation. Since the repeatability factor is so low, it's difficult to project how large it will end up being and how long something will take to build. Given that project managers and Management can't put their hands on the code and actually see what is being done, unrealistic expectations regarding timelines get generated.

• QA / QC is not emphasized as heavily as it could or should be for larger projects. This goes back to having looser interfaces between components, and not having rigid specifications for how components should work. That looseness allows for unintended consequences and for bugs to creep in.

• Consistently measurable qualifications. Generally, people speak of the number of years they've worked in X language or in programming. Time in is being used as a substitute for caliber or quality of skill. As has been mentioned many times before, interviewing and finding good programming talent is hard. Part of the problem is that the definition of "good" remains very subjective.

I don't mean to be all negative, and I think the software industry has made significant strides from where we've been. Forums like this and others have really helped promote conversation and discussion of design principles. There are organizations working to standardize on "baseline" knowledge for software engineers. There is certainly room for improvement, but I think the industry has come a long way in a reasonably short period of time.

Answer 2

The answer is surprisingly simple: those 'other industries' do have a high failure rate. We're just comparing the wrong things. Writing software is often called 'build', and so we compare it to the manufacturing or construction phases in other industries. But if you look at it, it's not construction at all: it's design. Software designs are written in code, and building is done by computers, whether by compiling software or directly interpreting it on the fly.

Many designs in other industries either take way longer than originally estimated, cost way more, or simply never see completion. Sound familiar?

So, what are we doing when we're planning software? Well, we're still designing, but at an earlier stage.

In software, there's no manufacturing line of note. Building the final component is (comparatively) cheap, and replication of that final product is both perfect and effectively free--you copy the build artifacts.

Answer 3

To point out some figures:

1. Change of requirements after implementation started; for example when the first Airbus A380 started to be created in the factory I cannot believe that if someone wanted 200 more seats, those would be put there; but in a large software project even after the programmers started development 5 more types of users can be added.
2. Complexity - large software projects are extremely complex; probably the most complex projects human kind designed and developed;
3. Not enough resources are spent in architecture and design phase;
4. Field immaturity - software engineering is relatively a young discipline compared with other engineering sisters; this has two implications: a) Not so many heuristics and good practices; b) Not so many very-experienced specialists;
5. Lack of mathematical proof - in most of the cases mathematical formal methods are not used to prove that a piece of software works as required; instead testing is used. This does hold true in other engineering fields which rely more heavily on mathematics; the reason of this is as complexity;
6. Rush - many managers have unachievable deadlines; so quality of code is placed second, because of the deadline.

Answering strictly to the question - I tend to believe that others have very high expectations (especially in delivery time) from programmers and do not understand exactly how difficult programming large software is.

Answer 4

The premise of the question is a bit flawed. Both the A380 and the Boeing 787 were delivered years late.

In the case of the A380 much of the delay was caused by the French and German units of Airbus using different and slightly incompatible versions of CATIA design software. This incompatibly manifested itself as wiring harnesses that didn't quite fit the airplane.

There wasn't anything wrong with CATIA, which is the most widely used aircraft design software, but someone somewhere dropped the software configuration ball.

The Boeing 787 also suffered from software related delays, but most of its problems were more traditional new airplane problems like weight control and delivery of substandard parts by suppliers.

Both the A380 and the B787 had to modify their wing designs after the initial aircraft had structural issues.

Large complex projects are difficult for humans in all fields.

Answer 5

Skyscraper guy here. Not sure if I can answer your question but I can surely shed some light into various items in the thread. Buildings do indeed occur very fast. A major constraint is locale (regulations). But in general it takes 3 to 10 years for a tall building from start to finish.

I think comparing a new building with a new software project is not very accurate. A new building is closer to a new version of a kernel or OS. In this respect software development is much faster. We never build from zero as this will be a high risk task the client would never sign up for. Most design and development in buildings is derivative of proven and completed projects.

From personal experience only one in ten projects ever get built. The process is development-driven rather than design-driven, so the moment something like the price of steel goes up the whole project is out the window, or designed, as design is the cheap component of the process.

Design takes a month for concept to schematic, two to six months to design development, another six months to details and construction documents by a team of architects, planning consultants, structural engineers, wind engineers, services engineers, quantity and cost consultants, surveyors, accessibility engineers and the list goes on...

The argument of virtual versus physical is not very accurate. We also work mainly with virtual tools, and moreover we are both time- and scale-remote from our final product. In most cases we can not even test aspects of buildings in mockup scale and we use simulation to try predict what may come about.

Complexity-wise there are differences, but overall it is maybe about the same. We not only have interrelated units and multiple levels of tiered abstractions and interfaces but also people are very much specialized in small parts of the process that make communication very difficult.

As for the argument of design versus development, I think both processes are design-built. It sounds academically nice to keep these separated but it is not possible to design if you don't know how things work. You just increase the risk of failure.

Overall my (potentially wrong) estimation as per OP's question is that programming is more of an art than engineering. Why would people take pleasure and even do it for free, find expression and elegance in it? Computer science is also (as on the tin) more of a science than engineering. Why would you try to prove algorithms instead of just patching existing parts together and work to make things just work? Not sure if this makes any sense; I'm not a software guy.

One aspect that strikes me with software design and development is about the medium itself. Computers make human-centric work very unnatural. Everything is so very explicit and there are few tolerances. It's hard to mentally work your way around this, and some get away with it by dumping complexity within. If nothing else this may have something to do with it?

Answer 6

Then how long did the design of those took? Year? Two? Ten years? The design is the most complex part of building something, the construction itself is easy.

Based on this article, it is slowly being understood, that software development is mostly design process where design document is the source code itself. And the design process is totally different from the production process. It requires experienced people and is impossible to plan, because even small requirement changes can result in huge changes in the overall architecture of the project. This understanding is the basis for agile methodologies that focus on improving code quality as the final design document and taking testing and debugging as parts of the design process, just like they test airplane models in wind tunnels.

The construction itself is handled automatically by compilers. And thanks to that, we are able to build whole products in a matter of minutes.

The reason why software projects are finished with huge delays and inflated costs is because managers still think they can estimate, predict and plan such a design process. This backfires more often than it is actually valid. They still think that by tying people into a rigid construction process they can somehow increase quality even though end result is mostly opposite with increased costs and missed deadlines.

Answer 7

Imagine, in the middle of the design of the Airbus A380, someone piped up in a meeting and said, "Heh, could build it as a triplane?" Others joined in saying, "Yeah, yeah. A triplane. More wings are better." The next thee years is spent turning the A380 design into a triplane. At another meeting, someone says, "A triplane? That's old. We want a biplane. Just remove one of the wings."

Or imagine, in the middle of a bridge construction project, someone says, "Heh, we just made a deal with a shipping company. They need the bridge to be another 40 feet higher, because their ships are much taller. Fix it. Thanks."

These are but some of the reasons why software projects, big and small, end in failure at an alarming rate.

Answer 8

As someone with a mechanical engineering background working in IT, I've often wondered about the reasons of the low success rate in IT.

As others in this thread, I've also often attributed the failures to the immaturity of IT, the lack of detailed standards (yes I'm serious, have you ever checked the standard sheet of a simple bolt?) and the lack of standardized components and modules.

Other industries, like building construction or ship building also have much more "beaten paths": knowledge and experience of a particular solution prototype, which - in customized form - is re-used again and again. Ever wondered about why buildings, ships or airplanes of different size and purpose somehow look so similar? (there are exceptions to the rule of course...)

That is because those prototypes are well researched, well understood, generally used and have a proven track record. Not because it couldn't be done any other way. In IT standardization is rarely the case: (large) projects tend to re-invent components, doing research and delivery at the same time and with the same people!

These inevitably lead to one-off products, which are expensive to develop and service, are error-prone and fail in unpredictable ways under uncertain conditions. And because of this, of course, these products are much quicker obsolete, written down and replaced at equally great costs with only slightly better ones. What IT needs is the equivalent of the industrial revolution, which turned middle-age artisans into efficient factories.

That said, there are factors that make IT truly unique however. As opposed to those other mentioned industries, IT is truly ubiquitous: it is used in every aspect of our modern life. So it's a small miracle IT achieved this much progress and is capable of delivering the results it does.

Answer 9

I'm afraid that I disagree with your statement.

Airbus and Boeing are two examples of companies that build planes. How many companies that build planes are there? Very few, if you would compare it to how many companies build software.

It is equally easy to screw an airplane project as to screw a software project. If only the entry barrier was so low in the aircraft-building industry as it is in the software industry, you will certainly see many failed aircraft projects.

Look at cars; There are high-quality manufacturers that build very durable and highly advanced automobiles (think Land Rover, Mercedes) and there are ones that build cars that won't last a year without having to repair them (think Kia or Cherry). This is a perfect example of an industry with slightly lower entry barrier, were you start to have weaker players.

Software is no different. You have lots of buggy products, on the other hand, Windows, Office, Linux, Chrome, or Google Search are very high-quality projects that were delivered on time and had similar quality level as an aircraft.

The other point that many people miss is how much maintenance goes into maintaining a car, a tanker or an aircraft that we just take as a fact of life. Every plane has to undergo a technical check-up before every take off. You have to check-up your car every several k miles and do so regular stuff like change oil, change tires.

Software needs that too. If only people spent as much time on diagnostics, prevention or auditing software's state and quality as they do with mechanical/physical products, I would expect way less statements like these. Do you read your application's logs each time before you launch it? Well.. if it was an aircraft you would have to ;-)

Answer 10

Digital building blocks can be 1 or 0. There is no inbetween.

A mechanical design has a level of tollerance. I can put one less than perfect rivet into a bridge and it will most likely will not fall down, however, in code even just once instance of putting a 0 where a 1 should be can fail the entire program.

Due to the logical and interative nature of computing, any, even very small changes, can lead to drastic failure.

Answer 11

I have often wondered the same thing. It certainly feels (occassionally) like we're a bunch of amateurs that don't have any idea what we're doing. I dislike explanations that put the blame on managers or other external factors -- we the developers should be responsible for what we create.

I think we are in a business where errors are cheap. Patching software is cheap, compared to rebuilding a skyscraper, or recalling every sold cellphone.

This has created a culture where bugs are a part of everyday life. They are accepted with a shrug. While some bugs are probably unavoidable, should they dominate our day to day work? I completely understand managers who don't feel that QA is worth the trouble, precisely because they expect bugs anyway. I don't understand programmers who don't make every effort to produce error-free code, because correcting bugs is boring as hell.

In essence I believe it is a culture problem, and I hope it will change.

Answer 12

Engineering standards and practices are very different in IT (as an independent industry) than in aerospace. This is perhaps most easily understood by considering how IT professionals react when encountering standards documents for IT in aerospace. For example, the Joint Strike Fighter C++ Standards that have made their way around the Internet in recent times.

Many express bemusement or wistful resignation (wish we could do that way); and many respond with ridicule, claiming there is no practical way to deliver consumer products in this way. This may even be right, given the expectations, not of consumers, but of management. There is a great deal of distrust for coders who just code and code for a few weeks, not demoing anything. God help the coder who merely designs something for two weeks. Not so with airplanes.

In software, people really expect to have something right now. Sure, the reasoning goes, it will take a little while to have it really solid; but can't we have some complex thing described in simple terms in a week? One learns, also, that complex things described in honest, complex terms rarely excite the imagination; and thus many engineers end up being complicit in an imagined world of really simple things being put together in creative ways (as opposed to a world of hard things being done really well).

Answer 13

Some stuff from me:

1- Standards and parts: They are plane manufacturers, not developers. I am not entirely sure, but my guess is that a lot of parts are outsourced. They don't build their own electronic/instruments, they get seats from some company, the engines are probably developed elsewhere, etc.

Software projects, on the other hand, almost always start from scratch with just some small frameworks/helpers in place.

2- Time to hit the market: Time is not a critical issue for planes. I bet the design of the Airbus was finalized years before it was finished, and they did chose to neglect any major breakthroughs that might happen in that time. (Same for car manufacturers, for example, the cutting-edge technology they develop at the moment will hit the streets in 5-10 years.)

For software you need to be very agile, if I start a huge project now and take three years to develop it without any change the chances are pretty high that I am relying on technology that is not available anymore or my product is completely outdated. This in turn offers a lot of problems.

3- Release-cycle and versions. - A plane needs to be completely finished when it is "released". There are no stable beta versions, nightly builds or similar. Additionally, once it's done, it can only be modified in a small way. You can't add an additional level with 100 seats to an existing boeing, it's just not possible.

Software on the other hand has incremental changes that are often just "builds that work", but not necessarily finished products. Also, in IT it's not unusual to say "hey, let's add another luggage compartment to our plane which holds additional 50 tons".

Answer 14

I think the answer is quite simple:

1) Physical construction and implementation have been around for as long as people have - we've had thousands of years to develop our methods and techniques for implementing physical projects. Software 'construction', which requires an entirely new and different skill-set, is no more than 50 years old - we haven't had enough time figure it all out yet.

2) Virtual construction is harder - you have to 'see' things in you mind that have no physical reality whatsoever. It requires you to analyze and abstract a lot of information before you even know what your product is supposed to look like and the steps it will take to create it. Not so when building a bridge or a building.

3) It's often much more difficult to find the source of a software failure or bug than it is when doing physical engineering. If a girder buckles, you see where it's buckling and you see the supports that are holding it and failing, etc. Finding a software defect can entail examining a great deal of code and interacting processes - difficult, time consuming, and not bound to the laws of physics and math in the way that physical structures are.

Answer 15

I will try answering using a verbatim copy of an article from Jack Ganssle's embedded muse. While this says firmware everywhere, just mentally replace it by software.

Compared to What?

Firmware is the most expensive thing in the universe. In his wonderful book "Augustine's Laws," Norman Augustine, former Lockheed Martin CEO, tells a revealing story about a problem encountered by the defense community. A high performance fighter aircraft is a delicate balance of conflicting needs: fuel range vs. performance. Speed vs. weight. It seems that by the late 70s fighters were at about as heavy as they'd ever be. Contractors, always pursuing larger profits, looked in vain for something they could add that cost a lot, but which weighed nothing.

The answer: firmware. Infinite cost, zero mass. Avionics now accounts for more than half of a fighter's cost. That's a chunk of change when you consider the latest American fighter, the F-22, costs a cool third of a billion a pop. Augustine practically chortles with glee when he relates this story.

But why is software so expensive? Tom DeMarco once answered this question with these three words: compared to what? He went on to discuss relatively boring business cases, but that answer has resonated in my mind for years. Compared to what? With software we routinely create product behaviors of unprecedented complexity. Sure, the code's expensive. But never in the history of civilization has anyone built anything so intricate.

Consider the following bubble sort, lifted shamelessly from Wikipedia and not checked for accuracy:

void bubblesort(int * A, int n){

    for(int i(0); i < n; ++i)

        for(int j(0); j < n - i - 1; ++j)

            if(A[j] > A[j + 1])

                std::swap(A[j], A[j + 1]);

}

It's a mere 110 non-space characters, perhaps tossed off in an hour or two. Suppose we didn't have software and had to implement a sort using some other strategy. What would it cost?

A mechanical engineer might boast that his profession built sorters long before computers. Consider IBM's 1949-era model 82 card sorter (http://www.columbia.edu/acis/history/sorter.html) with a throughput of 650 cards per minute, rather less than our code snippet might manage even on a 4 MHz Z80. The model 82, of course, only sorted one column of a card at a time; to completely sort a deck could take dozens of passes.

How long did it take to design and build this beast? Years, no doubt. And its functionality pales compared to our code which is so much faster and which can handle gigantic datasets. But that was 1949. How long would it take to build a bubble sort from electronic components - without FPGAs and VHDL, or a CPU?

In an hour I managed a rough block diagram, one above the chip level (blocks have names like "adder," "16 bit latch" and the like). But the sequencing logic is clearly pretty messy so I've just tossed in a PLD, assuming at some point it wouldn't be too hard to write the appropriate equations. And, yes, perhaps that breaks the no-programmable-logic rule, but to design and debug all that logic using gates in any reasonable amount of time is as unlikely as buck-a-gallon gas.

Assuming 16 bit words and addresses, the circuit will need around a dozen 16 bit latches, adders, and the like. Plus memory. And I have no idea how the unsorted data arrives into the RAM or how the results get exported. Those are unspecified design requirements. The software-only solution naturally resolves these requirements just by the act of writing the function prototype.

Translating the rough block diagram to a schematic might take a day. Then there's the time to design and produce a PCB, order and load parts (and change the design to deal with the unexpected but inevitable end-of-life issues), and then of course make the circuit work. We could be talking weeks of effort and a lot of money for the board, parts and appropriate test equipment.

All this to replace 7 little lines of code. Few real embedded programs are less than 10,000; many exceed a million. How much hardware and how much engineering would be needed to replace a real, super-sized computer program?

Consider a real program like a spreadsheet. How much circuitry would it take to make one without a processor? It could be the size of a city.

Firmware is the most expensive thing in the universe, but only because of the unimaginable complexity of the problems it solves. But it's vastly cheaper than any alternative. So when your boss irritably asks why the software takes so long, you know what to say. Compared to what?

So there! Software/firmware has unparalleled complexity.

Answer 16

Software engineering and management is fundamentally different than a lot of other engineering areas. The deliverables aren't physical, and the production process is the design and development process. Creating another copy of a piece of software has essentially zero marginal cost; all the cost is found in developing the first copy.

Because of the relative youth of software engineering and management as a discipline, there is some misinformation and falsehoods out that are still taken as fact (see this reference) which hinders software development and engineering as a whole.

Answer 17

Not all developers are created equally. Some are good, others are, well, not.

Try reading other people's code all the time to get a feel of what I'm saying. Too many extra logic statements can add risk. These risks can lead to ill behavior or bugs. Not enough logic statements and now you have null references. The good programmer understands this and knows when to do what and where. But no one is perfect. Things are complex. Add the poorly thought out work of others and it is easy to see how projects run away.

Answer 18

Cathedrals used to take up to 100 years to build.

Some Airbus airplane needs 1 million lines of code to work.

The more time you have been improving something, the more improvement you get, so give the software industry a couple of centuries of trial-error to get better, and we'll see how much it takes a to develop a solid huge project without bugs or flaws.

Answer 19

Large projects often occur in large organizations. If you've never worked in a large organization, there is one thing that is guaranteed to kill performance and productivity: bureaucracy.

Surprisingly, many people do not know what bureaucracy is (it is often confused with politics), or even if/when they have a bureaucracy problem.

We recently concluded a project to implement smart card authentication. It was originally estimated at three months. It took 15 months. There were not any cost, budget, scope, or technical reasons for the delay. The scope was actually quite narrow - only for accounts with elevated privileges (administrator accounts), about 1,200 total accounts.

Another significant factor is your business partners. This would include vendors. If your partners have a problem that introduces a delay in your project, there aren't many options that will actually fix the delay problem. They don't work directly for you, and you may not be able to fire that one person at a partner that may be the cause. The partner can be fired, or can be subject to financial penalties or disincentives, but that does not change the fact that the project has incurred a delay. This is precisely what occurred with Boeing when they undertook a mammoth sourcing strategy with the Boeing 787 Dreamliner.

Answer 20

I have a shorter version for you:

Whatever is easy to do, or streamlined, we write a program to do it instead of us.

And then fight with the meta-process instead.

It's not that much true, per se, but every day thousands of blogs are set up, instead of writing blog engines. Every workday, thousands of Excel macros are written, instead of writing specially-designed database applications for these.

There are a lot of other factors - some of them mentioned here - but I wanted to add this point to the discussion.

Answer 21

Lack of accountability... People get sued when an aircraft crashes. The software industry declines any responsibility in any software defect, therefore creating a lack of incentive to create a robust and safe product.

Answer 22

Most large projects have a high degree of separability of sub-projects, where you can define a small number of design constraints; the whole project will work when those sub-projects each are completed. If something goes wrong in a sub-project, the whole effort is not thrown into question; you just look for alternate ways to complete the sub-project (e.g. use a different engine).

This is possible but difficult, both practically and as a matter of human nature, in software projects.

In part, other industries have learned the hard way that this sort of separability is a good thing. For example, if you're going to use Rolls Royce aircraft engines, you do not need to use special Rolls Royce bolts and attachment points that only work with wings with a particular design, and then if you try to switch to Pratt and Whitney, you have to redesign your entire wing from the ground up (which, in turn, requires a complete redesign of the fuselage, which in turn requires you to buy different seats, which in turn requires you to buy a different in-flight entertainment system, which...). There may be a few linkages--you can't just swap engines without a care--but big projects generally work better when such things are minimized.

I postulate that big software projects designed as a cluster of small software projects with clean interfaces between each other will not fail particularly often, as long as the big project is actually solved by the cluster of small projects. (It is possible to make a mistake in this regard.)

Answer 23

Building software systems is very different from building physical structures. That is, the implementation is very much different. While for example building a huge tanker, you do lots of relatively simple (not easy though!) tasks, such as welding parts together by a robot or by hand, tightening all the nuts and bolts, painting, do the decoration by carrying in all the equipment and furniture and such. All of this is very simple stuff to do, really.

However, when it comes to software, it gets much more complex. For example, how exactly do you implement the secure login and user credential storing part of the product? What libraries and tools can you use? With what libraries and tools are you familiar with? How exactly do you go about writing the hashing + salting function and how do you ensure it is secure? You can do this in so many ways that it's impossible to set any actual practical design patterns for these kind of things. Yes, the said design patterns do exist on a smaller scale as "best practices", but every single software system is very different from the others, and the field advances and changes at so rapid pace that it's essentially impossible to keep up.

When building a house, you don't really run into such problems where you realize that the main supporting walls seem to be inadequate and need to be replaced, requiring you to demolish the progress so far and start from the base by redoing the support walls. You tackle such issues at the design phase, because it's relatively simple to predict what kind of support walls your building needs.

That is not the case with software though. You can't really design the whole product as a single entity and then implement it. The software design process is usually iterative, and the goals and requirements change as the product is being implemented and tested. Software development as a whole is an iterative process in which things usually change when least expected, and many times such changes impose challenges which require more work, more complexity and unfortunately and ultimately more money, time and hard work to get right.

So, in essence, the reason why it is hard to deliver big projects and estimate project timelines and roadmaps is that software development and especially working design are very complex fields. Complexity is the root problem.

Answer 24

The definition of "large project" is skewed.

A large project, technically, can be delivered on time, and flawlessly, granted it is something that's been built many, many times over the years.

• A Pac-Man clone.
• A calculator
• A text editor

I'm sure you're thinking..."but those are small projects! A text editor is simple." I would disagree with you. Computers are outrageously complicated. Just installing and setting up users on an operating system can be difficult at times, and you didn't even write the OS, or build the hardware.

The projects you're talking about are huge projects, akin to space exploration. How do you know how long it takes to develop inter-galactic travel? What model do we base it on? You have the known knowns, the known unknowns, the unknown knowns, and finally, the unknown unknowns, which happen to come up a lot in software development.

I think the problem is one of expectation. Just because the technology is there doesn't mean using it is going to be successful (or wise to use) for a while. If other industries behaved like the software industries did, we'd have black hole powered vacuum cleaners for sale by the end of the decade. Or some "visionary" would have the resources to build a moon base, and decide that a Starbucks would really "round out" the experience for visitors. I don't think the problem is the software industry, but the expectations placed on it.

Answer 25

Though it's hardly the only thing that could be mentioned, I think one basic thing is worth pointing out. Most products are intended to fit with existing behavior. Even a product that's a radical breakthrough (for example, the car) is generally built to fit with existing behavior, and simply make it a bit simpler/easier/cheaper/whatever to do that. Yes, there's often some side effect on existing behavior as well (for example, getting fuel for the car instead of food for the horses), but most of the latter tends to be a fairly minor side effect.

By contrast, almost any software that doesn't change the behavior of the users (for example, let them do their job considerably more easily) is basically guaranteed to be a complete failure from day 1. Worse, large software projects don't just involve the behavior of users on an individual level, but the behavior of large groups -- often the entirety of the organization.

In short, designing the software itself is often the easiest part of the job. The hard part is redesigning peoples' jobs for them. That's difficult to start with; doing it in a way that will not only work, but also be accepted is much more difficult still.

Answer 26

Airbus A380 was not a successful project as you have mentioned. I happen to work in a CAD/CAM company, and I was told that it (we had the Airbus prioject too) was delayed by few years, because they were using different version of software in different company. That is, different parts were being designed in different part of the world. And while integrating they came to know that all the design ca'nt be integrated, so they have to redesign it in one version. So looking at it I don't think it was successful. Had it came 2-3 years before, it would have been game changer for Airbus.

Also regarding robust software, you look at any airplane, car (ABS, EPS, climate control, etc.) or space shuttle they have more than 50% software which are running them and belive me they are very robust. It's just that we are more close to software, and there are many more software programs, so we see more errors in them.

Visit: http://www.globalprojectstrategy.com/lessons/case.php?id=23 and see how much successful Airbus A380 was.

Answer 27

Software engineer here, with an engineering background, and a wife who works in construction. We've had long discussions (and arguments) on the differences of our jobs.

Software engineering is about designing new things. Almost everything basic has been done in an open source library somewhere. In almost any job where a software engineer is hired, she has to design something that doesn't exist.

In something like construction and most forms of engineering, things that would otherwise be in a 'library' in software are already fully designed. Want to build a tower? Just copy and paste the plans from an existing structure, with a few modifications.

In fact, one of the main reasons I decided not to become an engineer is that you spend most of your time designing a 10% improvement to an existing invention, when that same time could be used to program something more visible, like a social network.

There are not many new designs in engineering; an extremely skilled engineer is someone who can manipulate an existing design into something new or improve on it. But almost every programmer is expected to modify designs, hack them, or create something new.

Look back at how far the standards have changed completely, from assembly to C to C++ to Java, JavaScript, C#, PHP, and so on. There's not a lot of code that can be recycled from 10 or 20 years ago. This is very different to say... the automotive or aeronautics industry when you can keep improving on designs from decades back.

Project management is notoriously difficult in software. Time estimates are best done by people doing the work, but when they're busy making estimates, they're not writing code. This tempts people to avoid any project management at all.

Often, a lot of code depends on specific people, and if these people are late or unable to perform, the code does not move ahead. In contrast, if you wanted to build a car, you can simply hire different people to assemble the tires, the chassis, the battery, the engine, and so on. Object oriented and existing frameworks makes this possible, but it may not be practical when you're designing everything from scratch.

Failures may be allowed in software. Testing can be costly. In software, it's very tempting to skip all that testing, when you can just fix a crash. In most forms of engineering, a 'crash' can be fatal.

You do have programming methods that use extensive testing, like extreme programming (which was actually used on software megaprojects). But with tight deadlines (that can be made tighter on purpose), it's tempting to skip all that programming and launch with bugs. The Joel Spolsky style of "always fixing all bugs" will save more time in the long run, but the undisciplined will skip this and fail.

Small projects are better. My wife once asked me to get a job in a big company. It ended up in an argument that big companies are bad companies... to her, a big company had a lot of resources, experience, functional project management, and the right procedures. To me, a big company is a dinosaur, where most of your time is spent on fixing code, testing it, and documentation.

I've seen million-dollar IT projects worked on by less than 10 people. More people would have slowed down the project and added unnecessary bureaucracy. WhatsApp is an example of a 'small' project that's worth billions of dollars. It's not that big projects aren't possible, but you simply don't need thousands of people to produce billions of dollars worth in software.

Answer 28

The difference in my opinion is the level of abstraction used. Each level of abstraction makes it harder to estimate the exact amount of time needed to complete a certain task.

Whereas a typical Airbus project is mainly using mostly already existing tools (well ok, maybe some tool occasionally isn't up to the task and needs modifications by the tool manufacturer), and creating a plan for how to use those tools to manufacture the particular model (which usually isn't all that different from the already existing models), with software projects this design of next generation Airbus would be mainly to write your main() function. That's not all the software engineer does.

Maybe an Airbus requires parallel development at different companies: for example the engine is not built in-house but rather in a separate company. However, in software engineering there's the problem that marginal costs of software are zero, so paying for something you need is almost always a bad choice (with some rare exceptions). Therefore, more of software tends to be built in house and for the rest standard open-source projects may be a good enough solution. In some cases, it may be found that commercially available tools are better than open source tools, but there's a limitation that due to difficulty in funding open source activities, a lot of what you might want may not exist -- and then you need to create it yourself.

With software, you often have to write entirely new libraries. In some cases, you may be able to find an existing library as open-source, but do you trust it more than you would trust a library written yourself? Do you save time or lose time -- if the existing library doesn't work well, you may even lose time to debugging. Is the existing library so intuitive that the 10-100 people who later have to read your code using that library immediately understand what it does?

Actually, using existing libraries is less predictable: whereas you can usually predict the amount of time it takes to write a new library (and the realization may be from anywhere between 0.5 to 2 times that prediction), with an existing library the realization could be between 0.001 to 5 times that prediction (with the worst case being you ditching that existing library and rewriting the damn thing yourself, and the best case being finding out the existing library is marvelously good). So, existing code actually doesn't decrease unpredictability, it increases unpredictability but on the average saves costs, so that's why it's the chosen approach very often.

Not only that, but the level of abstraction is usually higher. You sometimes build libraries -- in traditional engineering, that would be like designing your own power tools. But you don't do only that, you build libraries using libraries -- in traditional engineering, that would be like building robots that use the power tools you just designed. In fact, software engineering very often is more like building robots that assemble more complex robots that use the power tools you have designed. The level of abstraction is just staggering. Just take a typical stack trace that maybe shows 20 entries, maybe 50. In traditional engineering, having 20 stack trace entries would mean you have a tool used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot used by a robot. And 20 stack trace entries isn't even much in the software engineering world.

Sometimes, it has been said that software engineering will be automated away.

But who will build the automation tools then? That's the software engineer.

In fact, I would claim that this automation has made software engineering more productive. Every software engineer is all the time thinking about automating his own job and whether it saves any time -- the best person to automate a task is the person doing that task, assuming that person is capable enough as a software engineer. Unfortunately, this automation will increase the level of abstraction, so while it may save time on the average (assuming the software engineer making decisions has good judgement), but it also increases time schedule risks. Anything making the system more complex, while it may save time on the average, also increases time risks.

Also, sometimes traditional engineering projects get delayed. For example, Olkiluoto-3 nuclear power plant was supposed to be ready in 2009. The last time it was delayed was in 2022, and the latest schedule is after Christmas 2022 it should be running continuously. I think actually that Olkiluoto 3 has more in common with software projects (a huge unique project, a lot built for just this project), rather than with building airplanes. In fact, they are going to build nuclear reactors like airplanes, using manufacturing lines and building standardized designs, and making the things smaller so passive safety can be realized and less complex safety systems are needed. I wish it was that easy for software, too.

Answer 29

For me the main problem that software engineering face is use cases, user and cross platforms.

Use cases

How many use cases does an airplane have? Most of it is just to fly from one place to other. Maybe there are more such as radar, traffic control, etc., but the use case is clear and not much. In software engineering, we are faced with unclear requirements and user who do not know what they want. Different user need different configuration / flow, can an airplane be customized as user want (I want to have tv and refrigerator!)?

User

Who operates an airplane? A pilot, a copilot, some stewards (if counted) and tower operators. They are all pros and has their job descriptions clear. Software are used by noobs and dummies, not to mention evil hackers and crackers, while still need to be integrateable with other modules (such as OpenID or Google AdSense). If an airplane can be operated by dummies while still surviving from missiles or ninja robbers, then you can say that the airplane has the same quality with software.

Cross platforms

An airplane fly only in the earth's sky. I'm unsure about how they handle the foggy or windy or hot, cold and humid climate, but an airplane is not designed to fly at different gravitation level (I will be amazed if it can fly to Mars). Sometimes, an application must survive different platforms, such as Internet Explorer, Google Chrome, Firefox and Safari for browser (sorry Opera), or Windows XP / 7 / 8, or Linux for OS level. Not to mention mobile devices and different resolution and orientations.

Answer 30

One reason which hasn't really been covered in the other questions is that the field of software innovates at a speed which occurs only rarely in the material-based world.

One reason for this is that software evolution is a positive feedback cycle because software (programming languages, build tools, IDEs) is used to build software. It's the most obvious example for Ray Kurzweil's law of accelerating returns, resulting in progress at exponentially growing speed. Once you have mastered one framework, programming language, communication protocol it's time to move on to the next.

If airplanes were built like software we'd be changing the material with every other model, double their capacity and speed every 18 months, replace the engine technology for each new model with something just invented, while adding the ability to swim and search for extraterrestrial life.

Oh, and ideally it listens to voice commands and doubles as a fully immersive movie theater, paintball arena and night club with dark room once your work trip is over. The same thing! (The company which built airplanes which just got you reliably from A to B went belly-up a year after the one with the movie theater, paintball and nightclub feature came out.)