19

I'm not an expert programmer so this may be why, but I've come to notice that whenever I create complex code (like a Chess game I recently made), I am able to write the correct code to get the program working, although I find that later on- or even a few seconds after!- I often have to pause, and think, how does this work?

Not only that, but I also tend to not think about the code, and instead I just type away. For example, in my Chess game, I decided to use a five dimensional array for processing moves, and I found I could do this without too much conscious thinking. However, when I stopped and read over it, I found it challenging to get my head round the whole five-dimensional concept, and it did take me a few a minutes to fully understand what I did, and how the code itself works.

Is it normal for programmers when writing complex code to not understand what they are doing half of the time?

  • 13
    It is a sign that you are trying too hard to be clever. You need to write simpler, more modular code if you're having trouble reading it yourself. – SHODAN Nov 23 '14 at 16:53
  • 3
    To add to other (insightful) answers in the sense that it smells of over-complicated or poor designed code (do not worry, most of us had to pass that phase): Document. Both by giving proper names to variables/methods/classes/modules, by adding a proper description to each function, and by (when you see no other way around) writting a simple inline comment explaining why and how are you using some complex snippet of code. – SJuan76 Nov 23 '14 at 17:33
  • 4
    I never have 100% clarity over my own code. – CodesInChaos Nov 23 '14 at 17:53
  • 2
    That 5D array really sounds like it could use some abstraction. – user49272 Nov 23 '14 at 18:31
  • 3
    Do any developer have 100% clarity over his code at all times? – Johan Nov 23 '14 at 20:56
30

No, it is not normal 1. At least, it's not normal for good programmers. It probably is normal for someone learning to program.

Writing software isn't just slapping lines of code together until it works. You need to consciously work on making the code easy to understand. A programmer I respect highly once told me "code is read way more times than it is written". While that should be completely obvious, it was a fact that I hadn't been aware of until he told me. Much of your code is written only once, maybe rewritten once or twice more, but you'll end up reading the code often over the lifetime of the software.

If you're finding the code hard to understand minutes after writing it, that's a signal that your code is too complex. Stop adding code, and figure out a better way. For example, a five dimensional array is almost never a good idea. Even really, really smart programmers have a hard time understanding such a complex data structure.

The same programmer who told me about code readability also said "show me your data structures and I can tell you how your code works". Meaning, good code starts with clean, understandable data structures. If you design your data right, the code is almost a secondary concern. Admittedly, this statement is a bit of hyperbole because software is obviously more than just data, but it starts with data. So, work on creating clean, easy to grasp data structures and the code will be considerably easier to understand.


1 Certainly there is code out there that is very complex and hard to understand even by the smartest of programmers. Some problems are inherently complex. However, I would venture to say that the vast majority of code written by the vast majority of programmers is not that type of code.

  • 8
    [1] From a more pessimistic standpoint, it is normal but it's not good. – Dan Nov 23 '14 at 18:18
  • 1
    If the "five dimensional array" is merely a map from a 4-tuple or a 5-tuple to a strategy, the author could have used a hash table or a helper lookup function. However, if the author spends most of the time coding the mechanics of initializing the array (nested for-loops), then the actual "algorithm insight" is going to be drowned in the pile of "mechanical code". Programmers seek to keep the latter kind of noise separate. Thus, good programmers must know how to write libraries primarily to house those mechanical code. – rwong Nov 23 '14 at 20:28
  • 1-D array is a line, 2-d is a grid, 3-d is a cube, 4-d is a line of cubes, 5-d is a grid of cubes, etc. but I don't see a use for such a complex data structure when it comes to chess. – user2785724 Nov 24 '14 at 15:12
15

There are two kinds of this: 1.) confusion 2.) blissful ignorance

The first one is bad and may disappear with time and experience.

The second is a good one if projects become larger: If you have to remember every implementation detail just to be able to work with your code, there's something wrong with it (see "information hiding").

Every developer is going to forget how code was working, so he writes it in a way that another new developer would understand and be able to maintain it without breaking other parts of the programm which is unknown to him as well.

So "not knowing" is actually a constant in software development - it's just how or whether you manage it.

  • 1
    You're touching on something important here, which is how vital it is to program using common-sense patterns and conventions, because implementation details do indeed get forgotten. But if the patterns and conventions in the code are sensible, it is easy enough to pick the details back up from the context when necessary. On the other hand, if the code is all monolithic, inscrutable and overly clever, not only will you eventually forget the details, but other programmers trying to maintain the code will have a much harder time. – Craig Nov 23 '14 at 21:45
12

I'd say it's more common than people care to admit. Even Brian Kernighan alluded to it:

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.

When we drive to the store, we make a sequence of detailed adjustments to the positions of the pedals and the steering wheel. This is very easy in the moment. Now, imagine if you recorded those adjustments onto paper and gave them to a friend who needed directions to the store.

Similarly, we like to write code at one level of abstraction, but we like to read it in multiple higher layers of abstraction. Our favored way to write and read code is therefore in conflict. That means making code easy to read is usually a separate, conscious step, with a different set of skills.

What makes a good programmer is when they have difficulty reading what they just wrote, they create an abstraction at that point. A better programmer does that multiple times, getting pickier each time. Eventually, experienced programmers start off further along the process, but they often can still see room for improvement after reading what they just wrote.

  • I have to disagree with Brian on that one, I love debugging, I'm one of the few people I know in the industry who does but I don't rate code that I write that highly. – James Snell Nov 23 '14 at 18:28
  • @JamesSnell He didn't say that debugging is bad or unpleasant, merely that it is hard, and debugging complicated code is even harder. – cbojar Nov 23 '14 at 18:49
5

I think this is something that'll go away with experience.

If you're writing complex systems, you need to have the ability to write clean, maintainable code that both you in the future, and someone else in the future, can understand. So, in essence, the thing you're doing now isn't scalable.

You'll have lots of moments where you're looking at some code you wrote 6 months ago and thinking "what the heck is going on here?", but if it happens a day after you wrote the code, you gotta think 'clean code' more.

Source: never used a 5 dimensional array :)

  • 3
    @83457 - because a 5d array is a poor model of a 2d problem. If you really think it's good code then drop it over to codereview.stackexchange.com and see what answers you get. – James Snell Nov 23 '14 at 15:43
  • 2
    @83457 - If it IS 'confusing as hell' you've answered yourself. It might be possible that a 5-D array is not confusing but probably not for most of us, no matter our skill level. – dbasnett Nov 23 '14 at 16:06
  • 2
    @83457 being confusing as hell should be already a very good motivation to not use that. – Fabio Marcolini Nov 23 '14 at 16:06
  • 3
    As long as you have a good reason for each dimension, I don't see a reason to avoid a 5D array. Perhaps there is a better solution, such as a dictionary with a complex key or several lower dimensional arrays, but I can very well imagine a 5D array being appropriate for a tricky problem like a chess AI. – CodesInChaos Nov 23 '14 at 17:56
  • 2
    @CodesInChaos Except that those aren't just arrays, they represent meaningful sequences (for example nested decison trees). By naming them appropriately and giving them types so they can't be abused (even if those types are thon wrappers over arrays), you make the code clearer and less likely to contain bugs, for barely any cost. – deworde Nov 23 '14 at 18:04
5

"Normal" is very subjective, so I say: it is very common, but should be avoided.

One of the features of "the good code" (I heard such thing exists) is clarity: it should be as clear as underlying problems allows it to be. If problem is complex, the code would be complex as well, but that's inherent complexity, as opposed to accidental complexity (I first heard of this distinction in Rich Hickey's talk) introduced by mis- or not using the right tools, patterns, techniques and practices.

There are cases when lack of clarity is acceptable even when problem is not very complex, e.g. if you write a promo-site which you know is going to last as long as the marketing campaign goes. That's a throw-away code which must not be maintainable. For other (and most of) cases, it is not acceptable, because maintaining such code will cost a lot. Yet, it is common.

Related:

  • When Understanding means Rewriting - article about the problem of understanding the code.
  • Effective ML - a long talk about, amongst ML/OCaml, how to write the code for "readers" (i.e. maintainers): "Favor readers over writers." I recommend watching that no matter which language you use.
2

I don't think it is normal, but for very complex programs, like the chess program you mention, I think it certainly possible.

Many years ago, when I was just out of grad school (so I was still relatively inexperienced writing large programs), I wrote my first real compiler. The parsing was straightforward, but then I needed to target it for four different microprocessor instruction sets. I intended to write the compiler in its own language, so I first used just the features of the language I absolutely needed, wrote the first code generator in assembly language, and tested that it generated correct code for the subset of the language. I was then able to use the compiler to compile itself from then on, and add the remaining features and also use them in the compiler itself

I then wrote the backend code generators for each microprocessor, and tested that they all generated correct code, but while correct it was not very optimal. So I then proceeded to write optimizers for each code generator.

When I was testing the output of the code generators after adding all the optimization, I was frequently surprised at the code the compiler generated. It was often not the way I would have written the code by hand, but it was correct and pretty concise.

When it was not obvious to me how the code generator produced some of the code that it did, I tried to follow through the logic by hand but there were times I just gave up. If I had added a lot of trace output I would have been able to follow it, but I didn't bother since the generated code was satisfactory and I needed to get on to other projects.

  • Seems to me you have an extremely good education with a solid foundation, and that you have internalized the knowledge to a very high level. I guess this is somewhat rare among the typical programmers. Most typical programmers (including me) have to struggle to keep up with the knowledge needed for today's task, because the technology changes so rapidly. – rwong Nov 23 '14 at 20:36
  • @rwong Thank you. Most of it is experience -- I've been writing programs for 46 years and have no intention of quitting anytime soon. – tcrosley Nov 23 '14 at 21:09
2

There are a lot of decent answers here.

I have a couple of takes on this.

One is that if you don't understand why your code seems to work, then a) it probably doesn't (it probably only seems like it works), and b) you either didn't understand the problem domain well enough when you started coding, or didn't break the problem domain down into smaller, simplified units before you started.

My other take on this is that the real key is to use common sense patterns and conventions in your code. That's a far bigger topic than one little post can address. But look for good literature on the subject, including some of the old standbys like the books "Code Complete" and "Writing Solid Code."

Implementation details change. The only real constant is the functional objective of the code. If you ever write more than one function, you're going to forget specific, granular implementation details over time. But you should certainly understand how the pieces work when you construct them, and you should be able to draw a diagram of the whole and understand how the pieces work together.

If you use patterns, and follow common sense conventions, it will be far easier to pick the specific implementation details back up when you look at the code again. Or when somebody who has never seen the code before looks at it for the first time. Also, following those conventions and patterns over the course of time will mean that implementation details will stand out from the conventions and patterns themselves, which is another factor that will make the code easier to comprehend.

Most of the problems that we tackle with software are complex by nature. Software design is an exercise in managing complexity.

1

I wouldn't call it normal, but it can definitely happen. If it happens to you shortly after you wrote the piece of code in question, I guess that either your code is needlessly complex and should be simplified, or you're just easily distracted. :)

But if you put your code away, concentrate on other projects, and return to it after weeks, months or even years, it's not much of a surprise that you'll have to figure it all out again.

But there's something you can do about this. From what you say, it seems that you don't think enough about your code as you are writing it, and thus you're making it difficult for your future self to understand what's going on. Use that knowledge to your advantage: This is the best motivation for producing well-structured, well-documented, understandable code there is. You know from first-hand experience what happens when you don't take care of the quality of your code. Knowing that should make you a better programmer in the long run. When you collaborate on software projects, your colleagues will thank you for producing understandable code. And the defect rate of your code will improve as well.

Not the answer you're looking for? Browse other questions tagged or ask your own question.