Why do some programmers think there is a contrast between theory and practice? [closed]

Question

Comparing software engineering with civil engineering, I was surprised to observe a different way of thinking: any civil engineer knows that if you want to build a small hut in the garden you can just get the materials and go build it whereas if you want to build a 10-storey house (or, e.g., something like this) you need to do quite some maths to be sure that it won't fall apart.

In contrast, speaking with some programmers or reading blogs or forums I often find a wide-spread opinion that can be formulated more or less as follows: theory and formal methods are for mathematicians / scientists while programming is more about getting things done.

What is normally implied here is that programming is something very practical and that even though formal methods, mathematics, algorithm theory, clean / coherent programming languages, etc, may be interesting topics, they are often not needed if all one wants is to get things done.

According to my experience, I would say that while you do not need much theory to put together a 100-line script (the hut), in order to develop a complex application (the 10-storey building) you need a structured design, well-defined methods, a good programming language, good text books where you can look up algorithms, etc.

So IMO (the right amount of) theory is one of the tools for getting things done.

My question is why do some programmers think that there is a contrast between theory (formal methods) and practice (getting things done)?

Is software engineering (building software) perceived by many as easy compared to, say, civil engineering (building houses)?

Or are these two disciplines really different (apart from mission-critical software, software failure is much more acceptable than building failure)?

---

I try to summarize, what I have understood from the answers so far.

1. In contrast to software engineering, in civil engineering it is much clearer what amount of theory (modelling, design) is needed for a certain task.
2. This is partly due to the fact that civil engineering is as old as mankind while software engineering has been around for a few decades only.
3. Another reason is the fact that software is a more volatile kind of artefact, with more flexible requirements (it may be allowed to crash), different marketing strategies (good design can be sacrificed in order to get it on the market quickly), etc.

As a consequence, it is much more difficult to determine what the right amount of design / theory is appropriate in software engineering (too little -> messy code, too much -> I can never get finished) because there is no general rule and only (a lot of) experience can help.

So if I interpret your answers correctly, this uncertainty about how much theory is really needed contributes to the mixed love / hate feelings some programmers have towards theory.

Answer 1

I think the main difference is that with civil engineering, real world physics act as a constant, powerful reality check that keeps theory sane and also limits bad practices, whereas in software engineering there is no equally strong force to keep impractical ivory tower concepts as well as shoddy workmanship in check.

Many programmers have had bad experiences with runaway theory becoming an active impediment to getting things done (e.g. "executable UML", super-bureaucratic development processes). Conversely, dirty hacks and patches can get you pretty damn far, albeit slowly in the end. And as you observe in your last paragraph: failures are usually not as final and thus not as problematic.

Answer 2

Software engineering and civil engineering have little in common. Civil engineering efforts are limited by the physical properties of their materials and the environment. Civil engineers spend a lot of time learning about soil properties and material characteristics. Software development is physically limited only by the speed of the processors and the available storage. Both of these are easy to understand, and we don't spend much time studying them. The major limitation to software development is the human intellect. And no two developers are alike. Is this code maintainable? By whom? A three-line implementation of quicksort in Haskell may be obviously correct to some, but incomprehensible to others. A team of two may complete an application in a month, while another team of ten struggles to deliver in a year.

Software development is all design, the artifacts being designed are orders of magnitude more complex than any manufactured article, and each one is unique.

Answer 3

As a more-or-less honest-to-gosh mechanical engineer (with some civil) turned programmer, then CS (AI) PhD, then teacher, then programmer again (excuse me, Software Engineer), I've got a rant on this general subject.

In traditional engineering

• you gotta know your math and science because everything you do is based on it
• the "heroes" in the field are people who invent new things, discover new ideas, solve problems considered unsolvable

There is a "physics" that applies to software - information theory, but software engineers get little exposure to it, and certainly nothing applied. The most theory they get is computability and big-O.

Also I'm continually amazed at people who think knowing programming is enough, and they don't need to understand the subject matter of what their programs are about.

What's more, inventiveness is not encouraged. It is discouraged, in favor of least-common-denominator group-think methods, disguised as "readability". (Imagine if aeronautical or nuclear engineers were encouraged not to do anything that might be hard to understand for their junior peers.)

The things they do learn, like how to program web apps, are of great value. So is the skill of a plumber or electrician, but it is not engineering.

Answer 4

If I cut a corner on most software, and do something that's not the best design, but will get the job done, nobody is going to die. It's the same reason a hut in the garden doesn't need the same standards as a 10 story building. However, I can build a very large app like facebook, and if it screws up and loses some data, or whatever, it's not really that big of a deal. It's also simpler to fix the foundation of a large app after the fact, than it is to replace the foundation of a 10 story building. It all comes down to context, and calculating risk.

I can also, safely and simply keep adding to an app. You can't easily toss in a new third floor of a 10 story building (making it 11). I can toss in a new feature to a large app every day if I want to.

Now, good design makes all this easier in programming. But it's not impossible with poor design, and the risks, are...buggy software. Not usually death.

Answer 5

Bear with me on this one. I have a point.

I had a professor tell me once that procrastinating leads to more procrastinating, even though most people after a night of harried paper writing/cramming/programming say to themselves, "I'll never do that again. Next time, I'll start early and get done early." In my experience as the consummate procrastinator, I've found this to be true, and here's the professor's explanation why: no matter how unpleasant the experience of procrastinating is, in most cases, you get done having achieved relative success. This is high risk/high reward behavior. After a while, you forget about all of the unpleasantness, and only remember the reward. Thus, the next temptation to procrastinate is all the more enticing, because you succeeded the last time.

I think an analogy can be made here to the "get things done" programming technique we all too often see. A programmer or team of programmers, maybe out of ignorance, laziness, or perhaps a genuine time constraint, takes the "get things done" approach to programming, throwing all of your theory and maths and good practices out the window. And you know what? They get things done. It's not elegant, pretty, or maintainable, but it gets the job done. Maybe a non-technical superior who doesn't know a semicolon from a semaphore gives them some high praise for "getting things done". Thus, the next time the programmer is tempted to take this slack approach to programming, it's all the easier, because hey, it worked last time didn't it? It's the "easy" way out, unless your the poor, unfortunate soul who inherits such an application years later and has to maintain it.

I've been that poor, unfortunate soul, and so have many of you probably. I implore you all. Don't take the easy way out! :)

Answer 6

Your premise is flawed. The main reason civil engineers use engineering when designing large buildings, bridges, tunnels, etc. is to ensure that they are using minimum amount of material (concrete, structural steel, etc) which satisfies the required safety standards. It is entirely possible to build a tall building without much in the way of mathematics (e.g. the pyramids of the ancient Egyptian and Mayan civilizations) if costs of material and of labour is no object, but once built, there is usually no point in modifying them to make them use material more efficiently.

There is a somewhat different dynamic in designing large software systems. If anything, they are usually under-designed, but this is because the design can be changed dynamically as work proceeds, which simply cannot be done so easily with civil engineering projects.

The common factor is cost. Design on a traditional civil engineering project reduces costs (both actual, in terms of material, and potential in terms of liability), whereas there comes a point in software development where the cost of design increases beyond the value returned.

Answer 7

I would also point out, in addition to several other excellent responses that mankind has been doing the equivalent of "civil engineering" since easily the time of the Egyptians so we've had a lot of time to perfect the general theory of how things should be done. We've been building software for somewhere around 70 years or so (depending on what you consider the first "software"); I mean that we've not had the same amount of time to develop the same sort of body of experience.

Answer 8

An architect's/civil engineer's blueprints are virtually never identical to the "as built" plans. Something ALWAYS changes. Why? Because there are, and will always be, "unknown unknowns". There are things that you know and so can plan for, things that you know are unknown and so you can research and estimate, and then there are things that you don't know you don't know; "surprises". You aim to eliminate these in most systems by learning all you can, but all it takes is one little building code violation (which may be based on a rule that didn't exist 2 years ago when your building was being conceptualized) and your all-encompassing master plan has to change, sometimes quite drastically.

Software is very much like this; there's always an unknown unknown. However, unlike civil or structural engineering, software development is inherently much more tolerant of change based on the problems the unknown unknowns create. If you're building a 10-story building and you overestimated the load-bearing capacity of the foundation you put in your design, you either can't build the building to 10 stories or you have to tear out a significant amount of work to get back down to the foundation and reinforce or rebuild it. However, in software, if you underestimated the demands on a particular tier of the overall solution structure, there are many options for fixing that tier that don't involve invalidating all the other work. You can replace a single DB server with a more powerful one, or a replication/failover cluster, or a load-balancing/distributed cluster. Same for the webserver. If you coded an algorithm that is inefficient but simple based on faulty assumptions of input size, you can almost always simply remove and rewrite the implementation in a relatively surgical manner, without affecting other code that has knowledge of the algorithm (calls and passes input to it, or expects an output from it).

This relative ease of change allows a software engineer to code based on what he knows without worrying unduly about what he doesn't know. This allows for laxer application of theory and up-front conceptual design; you dive in and get it done, and along the way you find the things you coded that need to change, and change them. You must still know the concepts and theory, because when a problem is uncovered it's those things that will help you identify the cause and create a solution. But, you're allowed to make a snap decision without succumbing to "analysis paralysis", because if it turns out you made the wrong decision based on something you didn't know or didn't factor in to your "calculations", the mistake is easier to correct.

Answer 9

The difference is primarily because of the known requirements:

• On the theory side, everything is defined up-front, so you can know exactly what you need before you start.
• In practice, they're often not all there, or you discover something midway through the implementation that causes you to have to redesign something. So it's much better to jump with at least rudimentary designs, so you can discover these problems early on.

Additionally, when talking about "theory", it usually means the theory side of computer science, rather than software engineering. This is the part of computer science that's largely about finding better and more efficient algorithms, proving whether something is or is not possible (P and NP, for example), and so on. While it's good to have these in the back of your mind, they don't come up in software development very often.

We use libraries for that kind of thing as much as possible.

Answer 10

There are actually quite a few levels of software engineering depending upon what the software you are building is doing.

NASA needs software to control manned shuttles in space so naturally the level of engineering process is much stricter than that of building a website to show pictures of rockets.

One of my co-workers that worked for NASA previously described their software engineering process as writing hundreds of pages of justification and hundreds of hours of meetings to justify writing a single line of code!

Don't misunderstand me because I'm not trying to sound disrespectful when I say this but even after all that cost of time, resources, and billions of dollars the space shuttle still blew up.

Even civil engineers know that no matter how much theory they put into a design something will eventually break it so they also need to develop contingency plans.

When building software the cost of it crashing rarely causes loss of life so it is much easier to quickly throw stuff out there and test drive it. Let's agree that getting things done quickly results in weak code. Even if this is always the case, seeing software in action is the best way for a developer to see where it is weak and needs to be made stronger versus where it is weak and still many times stronger than it needs to be to keep up with the load.

To sum up, Premature optimization is the root of all evil or as my boss would always say Shipping is a feature!

Answer 11

Lots of good answers here, but I think the comparison between Computer Science and Civil Engineering is flawed.

Strictly speaking, what professional software developers do is more like Software Engineering than Computer Science. A better analogy is that Computer Science is the Physics for Software Engineering. Similarly, Civil Engieering is a collection of simplifications and approximations of Physics for practically building stuff.

I imagine that Civil Engineers rarely have to take into account general relativity when going about their job. Much of Civil Engineering can be safely built in Newtonian Mechanics. Similarly, Software Engineering can be accomplished very successfully with a roughly approximate understanding of theoretical computer science.

The big difference is that bridges, skyscrapers, and other products of Civil Engineering are reasonably well understood things. Software engineers are often building novel constructs, or using novel methods to build well understood things. Software Engineering is FAR less mature than Civil Engineering, and that will likely continue to be true for the foreseeable future.

TL;DR: Theory and practice are different in Software Engineering just like they are everywhere else. The proper analogy is Software Engineering : Civil Engineering :: Computer Science : Physics. But in practice, it is a little more complex than that :)

Answer 12

So my question is why do some programmers think that there is a contrast between theory (formal methods) and practice (getting things done)?

Building software is unlike building a bridge. In software, there are many objects to be built which may or may not be defined at the onset. Standards exist to increase ease of maintenance and developer collaboration, not to adhere to arbitrary mathmatical formulas or other such ideals. For example, when selecting behavior based upon a variable sometimes it makes sense to use a switch, other times a factory pattern. It depends on the ease of maintenance and identified pain points such as performance problems.

Another example can be made with data manipulation. It often makes sense to use delegates in the context of .NET. It is not so easy in Java because it does not have the framework support for functional programming style that .NET has. In other words, in the general case it simply is not possible to do X in situation Y. This is due to the fact that X and Y depend on N number of variable factors.

Is software engineering (building software) perceived by many as easy compared to, say, civil engineering (building houses)?

I'm not sure "easy" is the correct term. A lack of tangible evidence can lead to the perception that no work is being done. Or, similarly, that existing work is easily changed.

Or are these two disciplines really different (apart from mission-critical software, software failure is much more acceptable than building failure)?

Traditional Engineering and Software Engineering are very different for the reasons I already stated.

Answer 13

Your perception may be wrong here, or it includes many resources from people who haven't written sufficiently complex software.

Your experience is inline with what most people I know of (who have designed and written sufficiently complex software) would say.

That said, when it comes to most programmers, when the task of writing something gets to them the design ("the maths" as you put it) has already been done by the architect/lead/etc. before the task of writing it gets to them. So it may appear that way from the front-line level.

Answer 14

I think the reason for this contrast is that the life cycle of a software project and hardware or architecture project is different. Most software evolves gradually, it is not planned from the beginning to the end. Software developers can apply an iterative approach to development: plan, implement and listen to feedback. If the feedback is positive, carry on, it not - take a step back and reconsider your strategy. That's why software developers have things like agile development, minimum viable product and so on.

Civil engineers don't have such luxury. For them, once something is planned, you cannot change it as easily, as with software, because the cost of such changes can be dire. For software development, on the other hand, it's doesn't cost that much, and this can be used for their advantage.

But not every branch of software development can afford such approach. Making software, for instance, for aviation or medical services require very careful planning and a lot of prior calculations.

Answer 15

It seems the same to me. You build a large building out of standard blocks, standard strength concrete, standard steel. You build a big app out of standard libraries.

You don't try and mathematically formally prove a large app correct in the same way you don't try and write the wavefunction for a 100storey building

Answer 16

I was a mechanical and manufacturing engineer before I discovered some 20 years ago that my aptitudes lay in software. I sympathise with many of the elements which you have laid out.

I suspect that the true nature of the problem is about how we get things done. We've now got ten or so years of agile development under our collective belts, and the message is clear. Don't progress by layers; progress by features. Sure - there will be projects when you need to progress by layers (e.g. build your network stack before your web server) but for the vast majority of real-world projects, we have learned the lesson that delivering working features, one or a few at a time, is much more effective building huge untested theories, and then trying to implement them.

So let's take your hut example (I usually talk of making a bridge by throwing a log across a stream vs. a kilometer long suspension bridge... whatever!), and bring it to the world of software engineering. The main difference I see is that in software, most of the work is of a scale that it doesn't need big up-front modelling to succeed. The beginner's mistake is often to assume that things need more of this than they actually do, and for most of us, having made that mistake a few times, we're chary of making it again too often.

No argument - there are projects that need to begin with a committee of 17 software architects. In truth they are about as rare as 20 carat diamonds.

Answer 17

I think the analogy is flawed. As far as I'm aware, civil engineering does not have the same sort of theoretical basis as computer science; computer science was born from theoretical mathematics—like Turing machines. Civil engineering is about creating structures that resist mother nature and maybe even look beautiful. Again, I really don't know much about civil engineering, but I don't think there are civil engineer equivalents of P vs NP, the traveling salesman, and other fun things to bash your brains against. And there is definitely a place for our computer science theory—if someone solves the traveling salesman or the halting problem we are in for a lot of awesome new advances. But for a software engineer, whose business is to architect software, such problems are really only fun and games.

Now, I also think it depends on what you mean by "theory." Are we talking design patterns, or pumping lemma? Because having a good solid understanding of design patterns is absolutely critical to being a good software engineer. However, when architecting a large software system, theorizing about P/NP problems isn't useful. In that sense, I believe there is a very stark contrast between software engineering and theoretical computer science.

Or does theory refer to algorithms? You don't spend a whole lot of timing writing algorithms you learned in your algorithms class. Why? Because you typically only need them in particular cases (and then you look it up and research it), or you use a library already written for you. No need to write another bayesian classifier. Abstraction is an important principle in computer science. I think software engineers tend to not learn how an algorithm works until they need to.

Another reason is there are currently several popular "get it done" software development methods that are effective. For example, in agile development, you do not architect out an entire system beforehand. The reason for this is because you don't know exactly what you're building yet—you want what you're making to be flexible and adapt to new information and requirements. Designing it all out from the get go and then building just that does not always produce the best software. However, it is not the solution for everything. For example, say you're designing some distributed-computing-cluster-crazy-new thing. You can't do some napkin sketches and start your SCRUM.

TL;DR. I think there is some equivocation around the word "theory." Traditionally, theory refers to the theoretical mathematical aspects of computer science. Unless you are researching newer ways of computing, for the most part theoretical computer science plays no part in the day to day life of a software engineer. Software engineers care about design patterns and system architecture. Specific implementation details of certain algorithms are not important. Often times with less complicated ideas it is appropriate to not design a lot and just start coding. And I think this is where the idea comes from that programmers don't like theory.

Answer 18

The gap between theory and practice is too large at the moment. When doing theory, you are given three axioms and it is subsequently shown that a one-line theorem has a thousand page proof, or no proof at all. In software engineering, you're given inconsistent APIs of thousands of functions which give you a myriad of (bad) manners of implementing an underspecified feature.

Real software engineering would drive most of those in the formal field mad, and real mathematical software development drives those in engineering mad. Both fields require people of different aptitudes, and I don't think the aptitudes often overlap.

Answer 19

Formal theory assumes that you can accurately plan everything in advance like a manufactured product, that software will exist indefinitely within the same environment, and that solving a general abstract problem is always the goal. It assumes a 4D software-as-a-product lifecycle: design, develop, deploy, done. Formal theory is about solving the problem of software design using analysis, abstraction, generalization, and predicting future change. This is good if you have a well-defined problem in a straightforward domain that's easily analyzable, predictable, and fairly static.

Practical programming is about solving the right problem (not that of software design) in the right way right now, so that your coworkers can do their jobs better/faster/at all, or so that revenues can flow in to the company. Much software is not like a product, never 'done', but more like a living thing, that begins highly-specialized for one ecological niche, and may have a widely-variable lifespan during which it needs to solve new, unforseen problems in a wide variety of ever-changing environments. In the business world, with politics and legalities and competition and ever-changing organizations, structures, and trends, the requirements are often ambiguous, convoluted with all kinds of special cases, poorly-defined, and subject to rapid unexpected change. They're not analyzable, predictable, or static, and often not logical or reasonable. The software is as likely to be irrelevant in 2 weeks as to be still in use in 20 years. It comes into the world not knowing much or able to do much, and needs to be nurtured, groomed, and trained throughout its lifetime to grow up strong, flexible, and able to adapt to its ever-changing environments and new problems. If you neglect it after birth, it will go feral if it survives long enough, and cause pain and suffering, solving problems with blunt force.

Formal theory doesn't address the needs of much real-world business software. It tricks us into believing that software can be designed and done. That it's a product that can be occassionally fixed, polished, or have things tacked on, but not a living thing that needs to be raised properly with constant care and attention throughout its lifetime. So we end up with really ugly feral legacy code, but formal theory probably wouldn't have helped with that.

That all sounds pretty negative, but in reality I love using formal theory. A beautiful design always brings a smile to my face. However, that's mostly in my hobbyist programming that isn't subject to the vicissitudes of business. At work, I mostly deal with organic code and just hope that I can give it enough attention that it will grow up right, make me proud, and not be obnoxious and rude to others who have to deal with it.

Answer 20

The stakes are lower, the job is easier, and management rarely sees the value in good design. System instability, maintainability, and integrity are an "IT" problem--not a "Business" problem. All executives have one thing in common. They are either 95% focused on money, or they report to someone who is.

The rest of the battle is with your fellow programmers. Many of them cannot or will not commit to thinking about a problem before coding begins. Due to the above, a good deal of these people are senior developers, making it even more difficult to get a good design into production.

I have watched project leads waste years adding ad-hoc features and fixes to projects that were rocky to begin with, and then shoot down every attempt to bring order to the chaos with phrases like "too complicated" or "wasting time." It is not pleasant to watch a major project spiral to its inevitable doom because management will not admit they are building their own prison on a daily basis; however, I fear it is an unfortunate reality that many developers have witnessed and--for better or worse--learned from.

I try to find a medium in my work. I write no more code in "tainted" projects than is absolutely necessary, and I take every opportunity to move functionality out of them. "Between projects," I spend time on design and cleanup in the projects I actually have control over.

In the end, it is a big mess of politics and personal integrity that 75% of the world's programmers don't have the stomach for. I can barely stand it, myself.

Answer 21

First of all, I love this question. I've written like three 1000 word answers and they were all horribly wrong by the time I got to the end of them.

The problem with attempting to compare the two as analogous, I think, is that programming is a modelling process that can be as abstract or tightly bound to concrete as you want.

Structural engineering theory, on the other hand, is tightly bound to very specific sets of reality-based laws that you have to conform to. You can't just alter the context or the laws. The problem itself is rooted in those laws. In programming, however, sometimes the solution is actually altering the nature of the question or simply placing it in a different context.

Whether the MVC pattern, for instance, is a perfect fit, has a lot to do with that context. A desktop application typically deals in one language and one language only, not counting config files.

The front end of a web app on the other hand, consists primarily of two declarative (non-programming) languages and JavaScript. The one physical thing you can't entirely abstract away is the fact that there is always this http wall to chuck things over between server and browser. Regardless of how you bury it in code, that takes time and asynchronous design.

Obviously you can't use a popular and well-regarded pattern like MVC to handle front end concerns on the web exclusively without altering the way you might handle it in a desktop app context. In fact I would argue, you should be aware of what makes MVC useful but not even try to implement it in a particularly exacting or wholesale manner. The web app paradigm is unique in that all the look-at-me stuff is handled by the user's browser and all the data/model-ish stuff is typically on the server somewhere. But where does that leave the controller? All on the server or all on the front end? Somebody has to own that. Or maybe MVC isn't 100% the best fit for the scenario. Not a bad fit for .NET back end stuff. Not terrible in the context of specific UI widgets. But attempting to apply it to everything for the sake of consistency could be a bad move, IMO.

Building a house solves a problem. Typical programming problems, however, often involve solving problems within problems and sometimes the solution is to redefine the outer problem. Reality isn't especially keen on that idea unfortunately.

Answer 22

Glenn Vanderburg presents a great view on the differences between software engineering and more traditional engineering disciplines: http://www.youtube.com/watch?v=NP9AIUT9nos

If a civil engineer could test his designs without any costs before building the final thing he would make much less use of theory. If within seconds he could build a bridge a thousand times for free to test when it will break he would do so instead of spending months calculating when it might brake in theory...

In software development thats exactly what you do. Instead of calculating how fast your algorithm is in theory you can just test it and know the answer within seconds.

In fact most software today isn't limited anymore by physical constraints like computing power or memory. The limitation of software is the complexity that amounts in larger and larger systems. Its managing this complexity by keeping the system comprehensible by humans what makes the huge challenge in programming today.