How has an increase in the complexity of systems affected successive generations of programmers?

Question

As a "new" programmer (I first wrote a line of code in 2009), I've noticed it's relatively easy to create a program that exhibits quite complex elements today with things like .NET framework for example. Creating a visual interface, or sorting a list can be done with very few commands now.

When I was learning to program, I was also learning computing theory in parallel. Things like sorting algorithms, principles of how hardware operates together, boolean algebra, and finite-state machines. But I noticed if I ever wanted to test out some very basic principle I'd learned in theory, it was always a lot more difficult to get started because so much technology is obscured by things like libraries, frameworks, and the OS.

Making a memory-efficient program was required 40/50 years ago because there wasn't enough memory and it was expensive, so most programmers paid close attention to data types and how the instructions would be handled by the processor. Nowadays, some might argue that due to increased processing power and available memory, those concerns aren't a priority.

My question is if older programmers see innovations like these as a godsend or an additional layer to abstract through, and why might they think so? And do younger programmers benefit more learning low-level programming BEFORE exploring the realms of expansive libraries? If so then why?

Answer 1

it just isn't necessary because of the increased amount of processing power and memory available.

Having cheap memory, enormous disks and fast processors isn't the only thing that has freed people from the need to obsess over every byte and cycle. Compilers are now far, far better than humans at producing highly optimized code when it matters.

Moreover, let's not forget what we're actually trying to optimize for, which is value produced for a given cost. Programmers are way more expensive than machines. Anything we do that makes programmers produce working, correct, robust, fully-featured programs faster and cheaper leads to the creation of more value in the world.

My question though is how do people feel about this "hiding" of lower-level elements. Do you older programmers see it as a godsend or an unnecessary layer to get through?

It is absolutely necessary to get any work done. I write code analyzers for a living; if I had to worry about register allocation or processor scheduling or any of those millions of other details then I would not be spending my time fixing bugs, reviewing performance reports, adding features, and so on.

All of programming is about abstracting away the layer below you in order to make a more valuable layer on top of it. If you do a "layer cake diagram" showing all the subsystems and how they are built on each other you'll find that there are literally dozens of layers between the hardware and the user experience. I think in the Windows layer cake diagram there's something like 60 levels of necessary subsystems between the raw hardware and the ability to execute "hello world" in C#.

Do you think younger programmers would benefit more learning low-level programming BEFORE exploring the realms of expansive libraries?

You put emphasis on BEFORE, so I must answer your question in the negative. I'm helping a 12 year old friend learn to program right now and you'd better believe I'm starting them in Processing.js and not x86 assembler. If you start a young programmer in something like Processing.js they'll be writing their own shoot-em-up games in about eight hours. If you start them in assembler they'll be multiplying three numbers together in about eight hours. Which do you think is more likely to engage the interest of a younger programmer?

Now if the question is "do programmers who understand layer n of the cake benefit from understanding layer n - 1?" the answer is yes, but that's independent of age or experience; it's always the case that you can improve your higher level programming by understanding better the underlying abstractions.

Answer 2

I had ideas on this subject, and I put them into a book 20 years ago. It's long out of print, but you can still get used copies on Amazon.

One simple answer to your question is as old as Aristotle: Nature abhors a vacuum. As much as machines have gotten faster and bigger, software has gotten slower and bigger.

To be more constructive, what I proposed was that information theory, and its direct relevance to software, be part of computer science education. It is only taught now, if at all, in a very tangential way.

For example, the big-O behavior of algorithms can be very neatly and intuitively understood if you think of a program as a Shannon-type information channel, with input symbols, output symbols, noise, redundancy, and bandwidth.

On the other hand, the productivity of a programmer can be understood in similar terms using Kolmogorov information theory. The input is a symbolic conceptual structure in your head, and the output is the program text that comes out through your fingertips. The programming process is the channel between the two. When noise enters the process, it creates inconsistent programs (bugs). If the output program text has sufficient redundancy, it can permit the bugs to be caught and corrected (error detection and correction). However, if it is too redundant it is too large, and its very size, combined with the error rate, causes the introduction of bugs. As a result of this reasoning, I spent a good part of the book showing how to treat programming as a process of language design, with the goal of being able to define the domain-specific-languages appropriate for a need. We do pay lip service to domain-specific-languages in CS education but, again, it is tangential.

Building languages is easy. Every time you define a function, class, or variable, you are adding vocabulary to the language you started with, creating a new language with which to work. What is not generally appreciated is that the goal should be to make the new language a closer match to the conceptual structure of the problem. If this is done, then it has the effect of shortening the code and making it less buggy simply because, ideally, there is a 1-1 mapping between concepts and code. If the mapping is 1-1, you might make a mistake and code a concept incorrectly as a different concept, but the program will never crash, which is what happens when it encodes no consistent requirement.

We are not getting this. For all our brave talk about software system design, the ratio of code to requirements is getting bigger, much bigger.

It's true, we have very useful libraries. However, I think we should be very circumspect about abstraction. We should not assume if B builds on A and that is good, that if C builds on B it is even better. I call it the "princess and the pea" phenomenon. Piling layers on top of something troublesome does not necessarily fix it.

To terminate a long post, I've developed a style of programming (which sometimes gets me in trouble) where

• Invention is not a bad thing. It is a good thing, as it is in other branches of engineering. Sure it may be creating a learning curve for others, but if the overall result is better productivity, it is worthwhile.
• Haiku-style minimalist code. That especially goes for data structure design. In my experience, the biggest problem in software these days is bloated data structure.

Answer 3

High-level abstraction is essential to achieving ongoing progress in computing.

Why? Because humans can only hold so much knowledge in their heads at any given moment. Modern, large scale systems are only possible today because you can leverage such abstractions. Without those abstractions, software systems would simply collapse under their own weight.

Every time you write a method, you're creating an abstraction. You're creating a bit of functionality that's hidden behind a method call. Why do you write them? Because you can test the method, prove it works, and then invoke that functionality any time you want just by making the method call, and you don't have to think anymore about the code that's inside that method.

In the early days of computing, we used machine language. We wrote very small, bare metal programs with intimate knowledge of the hardware we were writing them for. It was a painstaking process. There were no debuggers; your program usually either worked, or it crashed. There was no GUI; everything was either command-line or batch process. The code you wrote would only work on that particular machine; it would not work on a machine with a different processor or operating system.

So we wrote high-level languages to abstract all of that detail away. We created virtual machines so that our programs could be portable to other machines. We created garbage collection so that programmers wouldn't have to be so diligent about managing memory, which eliminated a whole class of difficult bugs. We added bounds checking to our languages so that hackers couldn't exploit them with buffer overruns. We invented Functional Programming so that we could reason about our programs in a different way, and rediscovered it recently to take better advantage of concurrency.

Does all this abstraction insulate you from the hardware? Sure it does. Does living in a house instead of pitching a tent insulate you from nature? Absolutely. But everyone knows why they live in a house instead of a tent, and building a house is a completely different ball game than pitching a tent.

Yet, you can still pitch a tent when it is necessary to do that, and in programming, you can (if you're so inclined) still drop down to a level closer to the hardware to get performance or memory benefits that you might not otherwise achieve in your high-level language.

---

Can you abstract too much? "Overtake the plumbing," as Scotty would say? Of course you can. Writing good API's is hard. Writing good API's that correctly and comprehensively embody the problem domain, in a way that is intuitive and discoverable, is even harder. Piling on new layers of software isn't always the best solution. Software Design Patterns have, to some degree, made this situation worse, because inexperienced developers sometimes reach for them when a sharper, leaner tool is more appropriate.

Answer 4

Really good training involves both extremes, as well as a bridge between.

On the low-level side: how a computer executes code from the ground up*, including knowledge of assembly language and what a compiler is doing.

On the high-level side: general concepts, e.g. using associative arrays, closures, etc. without having to waste time worrying about how it works under the hood.

IMHO everyone should have experience with both, including their drawbacks, and a taste of how to get from low-level concepts to high-level concepts. Love associative arrays? Great, now try using them on an 50-cent embedded processor with 1kB of RAM. Like writing fast code using C? Great, now you have three weeks to write a web app; you can spend your time dealing with data structures and memory management using C, or you can spend your time learning a new web framework and then implement the web app in a few days.

As far as the complexity aspect of it goes: I do think it is too easy these days to make complex systems without a clear understanding of the cost of doing so. As a result, we have, as a society, built up a huge amount of technical debt that bites us from time to time. It's like earthquakes (just the cost of living near a geological fault, right?), only it's gradually getting worse. And I don't know what to do about it. Ideally we'd learn and get better at managing complexity, but I don't think that's going to happen. A responsible engineering education needs to include a lot more discussion of the consequences of complexity than most of our universities are currently providing.

---

* and, anyway, where is the "ground" in how a computer executes code? Is it assembly language? Or computer architecture? Or digital logic? Or transistors? Or device physics?

Answer 5

I feel that high level programming has many advantages and is an essential part of a programming language. One of the reasons why Java become successful was that it has a comprehensive library. You achieve more with less code - just call a predefined function.

We can now distinguish programming language users from programming language writers (compiler writers). We leave the optimisations to compiler writers. We focus more on maintainability, reuse, etc

Answer 6

The increase in the complexity of systems is relentless, oppressive and ultimately crippling. For me as an older generation programmer, it is also bitterly disappointing.

I've been programming for well over 40 years, having written code in 50-100 different languages or dialects, and become expert in 5-10. The reason I can claim so many is that mostly they're just the same language, with tweaks. The tweaks add complexity, making every language just a little different.

I have implemented the same algorithms innumerable times: collections, conversions, sort and search, encode/decode, format/parse, buffers and strings, arithmetic, memory, I/O. Every new implementation adds complexity, because every one is just a little different.

I wonder at the magic wrought by the high flying trapeze artists of the web frameworks and mobile apps, at how they can produce something so beautiful in such a short time. Then I realise how much they don't know, how much they will need to learn about data or communications or testing or threads or whatever before what they do becomes useful.

I learnt my craft in the era of fourth generation languages, where we genuinely believed that we would produce a succession of higher and higher level languages to progressively capture more and more of the repetitive parts of writing software. So how did that turn out, exactly?

Microsoft and IBM killed that idea by returning to C for writing apps for Windows and OS/2, while dBase/Foxpro and even Delphi languished. Then the web did it again with its ultimate trio of assembly languages: HTML, CSS and JavaScript/DOM. It's been all downhill from there. Always more languages and more libraries and more frameworks and more complexity.

We know we should be doing it differently. We know about CoffeeScript and Dart, about Less and Sass, about template to avoid having to write HTML. We know and we do it anyway. We have our frameworks, full of leaky abstractions, and we see what wonders can be done by those chosen few who learn the arcane incantations, but we and our programs are trapped by the decisions made in the past. It's too complicated to change or start over.

The result is that things that ought to be easy are not easy, and things that ought to be possible are nearly impossible, because of complexity. I can estimate the cost of making changes to implement a new feature in an established code base and be confident I'll be about right. I can estimate, but I can't justify it or explain it. It's too complicated.

In answer to your final question, I would strongly advise younger programmers to start as high on the layer cake as they possibly can, and only dive down to the lower layers as the need and desire provide the impetus. My preference is for languages with no loops, little or no branching and explicit state. Lisp and Haskell come to mind. In practice I always finish up with C#/Java, Ruby, Javascript, Python and SQL because that's where the communities are.

Final words: complexity is the ultimate enemy! Beat that and life becomes simple.

Answer 7

My question though is how do people feel about this "hiding" of lower-level elements. Do you older programmers see it as a godsend or an unnecessary layer to get through?

Neither, really.

Layering is necessary because without it, you reach a point where your system becomes unmaintainable spaghetti. It's also one of the tenets of reusability: if the developer of a library did a good job, the people using it shouldn't have to care about the details of the implementation. The amount of canned code we use in our systems has grown by orders of mangitude from what it was when I wrote my first program 35 years ago. That growth means we're able to do more powerful things as time goes on. This is good.

The place where it's been a problem for me is entirely cultural. My pragmatic half understands that it's no longer possible to wrap my mind around every last detail and be able to finish the things I want to get done. (Getting older doesn't help, either.) My cantankerous graybeard half had a difficult time letting go of many years of having such a fine-grained understanding of everything I worked on.

Do you think younger programmers would benefit more learning low-level programming BEFORE exploring the realms of expansive libraries?

As has been pointed out in other answers, there's a balance to be struck between attracting and maintaining the attention of neophytes and giving them an ideal, from-the-bottom-up education. If you can't do the former, the latter can't happen.

I see things in our industry that parallel the rest of society. It used to be that almost everybody grew their own food and spent a lot of time doing that. Since then, we've sprouted specialists called farmers who do that job, freeing up others to do other things that contribute to society. I buy my food at a grocery store and would be thoroughly unable to produce most of it on my own if I had to. We have a similar thing going on, albeit on a much more compressed time scale. Programmers are specializing in some set of layers and not others. The average guy writing GUIs may know that there is such a thing as swap space but probably doesn't know or care much about how the operating system is managing it.

The upshot of all this is that it's no longer about just development. Continued specialization means developers will need to continue improving their communication and integration skills.

Answer 8

As with everything, a little bit does you good, but too much hurts. The problem is too many systems don't know when to stop - just 1 more abstraction, to help you program faster... but then you end up coding in the real world where things are never quite as simple as you want, and you spend more time working round the edges than you would have spent with a less featured abstraction.

Its ably described here

or here - "with a single line of code you could add 500 users to the domain"...

Your abstractions try to hide the complexity from you, but really all they do is hide that complexity. The complexity is still there, its just that you have much less control over it - and that's why you end up with this kind of situation.

Answer 9

Do younger programmers benefit more learning low-level programming BEFORE exploring the realms of expansive libraries? If so then why?

I don't think so. There are still plenty of situations in where it's beneficial to be aware of low the 'layer below' works, e.g.

• When debugging a problem on layer n, it's can often be explained by considering what happens on layer n-1 (i.e. the layer below). I guess layer 0 would be "transistors" but if you want to explain a problem with transistors you would probably talk about physics (e.g. heat), so maybe physics is really level 0.

• When optimizing code it (unfortunately) does help at times to lower the level of abstraction, i.e. implementing an algorithm in terms of a lower-level layer. However, compilers became really good at doing this for you if they actually see all the code involved. This reason became more popular recently with the boom of mobile and embedded devices, which tend to have weaker processors and where "performance per Watt" is much more relevant than on, say, desktop systems.

In general however, it became a lot easier to make computers do stuff (even if in slightly inefficient ways) which means that there are a lot more programmers than there used to be. This in turn made the "human" factor much more important: Robert Harvey's answer already mentioned that "humans can only hold so much knowledge in their heads at any given moment", and I think that's a very relevant aspect nowadays.

A major motivation in programming language and library (i.e. API) design is to make things easier on the human brain. To this day, everything still gets compiled down to machine code. However, this is not only error-prone, it's also notoriously hard to understand. So it's very desireable to

• Have the computer help you in finding logical errors in the programs you write. Things like static type systems or source code analyzers (I hear Eric Lippert works on a fairly popular one these days) help with that.

• Have a language which can be processed efficiently by a compiler and which communicates the intent of the programmer to other programmers to make working on the program easier. As an absurd extreme, imagine writing programs in plain english was possible. Fellow programmers might have an easier time to imagine what's going on but still, the description would be very hard to compiler into machine instructors, and it's notoriously ambiguous. So you need a language which a compiler can understan but which is also comprehensible.

Given that a lot of (most?) compilers are still very much general-purpose, they feature a very generic instruction set. There's no "draw a button" instruction or "play this movie". Hence, moving down the abstraction hierarchy makes you end up with programs which are very hard to comprehend and maintain (though trivial to compile). The only alternative is to move up the hierarchy, leading to more and more abstract languages and libraries.

Answer 10

"if older programmers see innovations like these as a godsend or an additional layer to abstract through, and why might they think so?"

I've been programming since I was in high school, about 34 years, starting with Basic and Z80 Assembler, moving to C, various 4GL languages, Scheme, SQL, and now various Web languages. The scope, scale, and depth of the problems addressed by the profession experienced an inflationary period over that time, particularly in the 1990's. Constructs such as libraries, frameworks, and OS services are all contrivances meant to address the complexity that goes along with the expanded space of problems. They are not a godsend nor a burden in and of themselves - just a continued exploration of a vast solution space.

But, IMHO, "innovation" is better understood in terms of novel forms, and not mistaken for sideways movement - reintroducing forms we've already seen introduced. In some ways, the fecundity of an ecosystem suffers when the primitive forms don't compose, when they fixate decisions made early in the evolution, or can't reprocess their own detritus. Some, if not most, of the constructs we remain focused upon do not prioritize long-term sustenance of value as a concern. That has started changing - approaches like Service Orientation and Domain Driven Design, not to mention hypertext and graph-based models, for instance are altering the landscape. Like any ecosystem, eventually the dominant forms will give way to new forms; we are best served by allowing diversity, broadly and in small increments rather than large-scale top-down standardization followed by all-at-once collapse.

"And do younger programmers benefit more learning low-level programming BEFORE exploring the realms of expansive libraries? If so then why?"

I would argue that most human language is based on metaphors long since forgotten, so while I'd support learning low-level programming from a scientific/numeric literacy standpoint, it is more important that we seek primitives that will support the scale and scope of the problems we are tackling in a way that we can safely ignore the lower level of detail. A framework isn't a primitive, nor is an OS or a library - they are fairly poor at doing the kind of abstraction we really need. Real progress will take people who (a) know what went before and (b) can think in a novel enough manner to come up with something different enough to explore a space of solution that hasn't been explored before or was explored and forgotten.

OTOH, even if your goal is to work as a technician/mechanical level, some level of low-level programming exposure is still going to be helpful to developing your problem solving skills.

Answer 11

My first program (as a 15 years old teenager) was in 1974 in PL/1 on punched cards for an IBM 370/168 mainframe. My father was working at IBM and I was lucky enough to be able to go to the datacenter on Sundays.

At that time, a program of several thousands of statements (i.e.punched cards) was a big program (and a heavy one too, since many thousand of punched cards weighted many kilograms). Visual interfaces did not exist (a typical program read from its "standard input" using a punched card command starting with //GO.SYSIN DD * IIRC, but I did not master JCL). Algorithmics was important, and IIRC the standard library was quite small by today's standard.

Today, programs of several thousands lines are generally considered small. For example, the GCC compiler has more than ten millions lines of source code, and nobody is understanding them fully.

My feeling is that programming today is quite different from the 1970s, because you need to use much more resources (in particular, existing libraries and software frameworks). However, I guess that people developing datacenter software (e.g. search engines at Google) or some embedded software care as much about algorithms and efficiency than the average programmer of the 1970s.

I still think that understanding low-level programming is important even today (even if most programmers won't code themselves basic container algorithms like balanced trees, dichotomically accessed sorted arrays, etc...) because understanding the whole picture is still important.

A major difference between the 1970s and today is the ratio of cost between developer's (human) efforts and computer power.