Does software reuse preclude process repeatability

Question

Code Reuse as a Problem

I was thinking about this question on software delivery, and I kept coming back to the issue of repeatability and / or reproducibility. They matter, because if you don’t repeat a project then it becomes more difficult to improve the process you used to build the project. Engineering involves constantly improving the processes involved with design and construction in order to produce higher quality projects.

Software can rely heavily upon reuse due to its digital form. Instead of rewriting a module, we just call it again or copy it to the other system. Some examples are authentication / login or perhaps a logging function. There are many well known examples for those categories, and conventional wisdom is to reuse what exists instead of rolling your own.

---

Some Comparisons to Other Disciplines

Construction

In contrast, construction of physical systems (buildings, bridges) is nowhere near as reusable. It’s true that the blueprint of a house can be reused many times to build the same copy of the house, but the construction must be performed each time. Cut & paste doesn’t work like that in the analog world. Bridge blueprints are less reusable that houses because site conditions will vary.

Master builders are experts recognized for having designed and / or built tens, hundreds, or thousands of things in their area. For example, Frank Lloyd Wright, a world renowned architect and designer designed more than 1,000 structures and completed 532 works. Contrast that with Anders Hejlsberg who has designed “just” five languages (Turbo Pascal; Delphi; J++; C#; Typescript). In many ways, it’s an unfair comparison because the domains are different. But at a broad level, the quantifiable production from two very intelligent people is vastly different.

Martial Arts

Martial artists will say that mastery of a move comes only from thousands of repetitions. After a good portion of those repetitions have been put in, many martial artists are surprised at how a previously perceived to be complex kata or form has become simple. Instructors of those students will also notice how the motion becomes more fluid and purposeful as well as having an economy of motion. Likewise, experienced martial artists are able to pick up more complex katas more quickly than less experienced students. Experience from repetition has given them a framework or process that allows them to learn more quickly.

Woodworking

Woodworkers experience a similar transformation. Hobbyist woodworkers always refer back to their first project that required a lot of drawers. If they complete the project, they gain a new appreciation for the efficiencies that assembly lines produce. There are other benefits such as a better understanding of how to lay out the drawers parts on the sheet stock in order to maximize use of the wood. Compared to hobbyists, professional woodworkers are able to more quickly design, start, and construct items that they have made many times before. They also gain an ability to see inherent issues within someone else's design having made that mistake in their work.

---

So, does software reuse prevent software developers from becoming more proficient?

In many ways, software design and construction is always new. We don’t repeat past works, because if we can reuse a module, library, or system then we do. We’ll preferentially extend an existing system before rewriting the entire thing from scratch. But repetition is what allows us to find efficiency in the design and the construction. Anyone who has practiced a sport or physical activity will tell you that repetition is the key to becoming a good practitioner.

My question: Does software’s ability to be reused prevent the necessary process improvement and efficiency that comes from repeating a project?

Answer 1

The ability to reuse software does not prevent process improvement.

If you think about the processes that go into building software - developing requirements, designing the system, implementing the system, deploying the system, managing requirements, managing configurations, verifying and validating work product, tracking changes, and a number of others (see the CMMI process areas for one possible breakdown of key activities in software development) - these are repeated on every project regardless of how much reuse you have. In addition, each has some kind of quantitative and qualitative measures that can be used to determine how good that particular process or activity is, and as a result, how good the development process as a whole is.

In one end of the extreme, we can assume a robust software product line. At the other, you can assume greenfield development. There's still a need to perform all of these processes, to varying degrees, although they may happen at different rates or perhaps even in different sequences. For example, in a high amount of reuse, a greater percentage of the allocated time may be spent on integration and verification/validation activities at a system level (requirements V&V, integration tests, system tests, acceptance tests). With new development efforts, a greater percentage of the time may be required at design and implementation. As long as you perform a process at least once during the course of a project, you can measure it (quantitatively and qualitatively). Once you make adjustments, and see how those adjustments impact some measure of either the process area or the overall capability to deliver software, and then refine the process for other projects.

They key to process improvement is to have some kind of logical breakdown of your activities and processes, determine how to measure them (preferably consistently), and how to understand those measurements to make process changes toward some end. It's not about repeating the project, but about consistency in how you repeat the process.

Answer 2

I think the idea that other engineering disciplines don't make use of reuse is wrong. Even when designing buildings/machines you still have components that are used by many other projects. For example, do you design you own screws? Engines? Doors or windows? Of course not. Those are often designed by different people who then use them in different product. And they are quite often standardised, which promotes even more reuse.

I think the problem likes in complexity. You simply cannot compare complexity of even the most complex buildings to complex software. It is a generally accepted idea that software complexity is what makes it hard to approach from the engineering side. The moment you have a process in place that allows you to create software of acceptable quality, you find that the complexity of software you need to create jumps in order of magnitude. So the process cannot be used. So if we had to repeat some part of software multiple times until we are satisfied with the result, we would never finish that software.

That is why clean code is promoted. Ability to change past code based on new experiences can be said to be form of design reuse. So instead of creating different software multiple times, we refactor and refine single piece of software by reusing new experiences and design on old problems. All while trying to make software do the same thing.

Answer 3

Software is different than most other disciplines, so the economics of where we best spend our time is often different.

In construction, you spend a certain amount of time and money on a blueprint (and software is far more like producing a blueprint than like building a building), then, roughly speaking, a whole lot more on actually building it one or more times. So it is worth it to put quite a lot of work into getting the blueprint right. More specifically to your question - it's worth repeating the effort of doing it over from scratch to make the end product a little better.

In software, when you have the blueprint, it is much cheaper to build the product than it was to make the blueprint. At least most of the time - if the software will be embedded in a pacemaker you are much closer to the situation of a bridge builder in some ways. But in general, reusing software might save 90% of the cost of your biggest budget item, vs. saving 90% of a much smaller budget item for building a bridge. So, re-use wins a lot more often.

As far as productivity - when you build, say, a bridge, you face really significant real world constraints. Imagine if architects were paid large sums of money to design bridges for massive multiplayer online games, where construction costs were near 0 and limitation significant less than the real world. They would design bridges that are freakishly complex by real-world bridge standards. The blueprint phase might take a little longer.

Also, there are a limited number of bridges to build, and, since design is a smallish part of the cost, you can pay for the best, and a few of the best can do most of the design. There are hundreds of thousands of software developers, and basically all of them have a giant backlog of things they would do if they had time. You aren't going to find one guy who does a massive portion of all that - it's kind of surprising that there are people who sort of come close, really.

Your real point seems to be that we may be losing something by re-using instead of trying to repeat and improve things. I think you have a point. The problem is that though it would most likely be more globally efficient to rewrite some of the foundational stuff and try to improve it, whoever takes that on gets all the risk and probably not that much of the reward. (There is also a huge practical problem of dependency hell, which probably takes some of the win out of rewriting things, but not to the point that it wouldn't be worthwhile, at least looking at the global picture. Copyright and patents also may force a proposed re-engineering effort bite off quite a bit of rewrite work to redo smaller pieces of existing code).

In terms of learning from repetition - in all disciplines this happens less in design than in construction, because there is less repetition, so less chance to learn, and perhaps less benefit. Also, the process of design probably just isn't that repeatable. It's a bit like having a process for writing a novel. A good process can almost certainly help, and software is generally far more collaborative than a novel, but repeating a process when the goal is to invent something new is problematic. Even novelists learn from the past, very much so, but a repeatable process is a secondary factor for creative endeavors. And if any part of software development is really truly repeatable, why isn't a computer doing it?

Answer 4

Does software’s ability to be reused prevent the necessary process improvement and efficiency that comes from repeating a project?

I have worked as a systems and software engineer in the same large project for the past 17 years, incidentally (thinking of the Airbus A380 reference in your first link) in the aircraft industry, though my responsibilities lie in the military aircraft sector. Stories like that are basically pure fiction, and actually really funny to watch when you have insider insight.

But for your brief and concise question: From my experience, I would say both yes and no.

Let me first say that I am all for software recycling in all forms (well, maybe not all...). The advantages of reusing just about anything, from cut-and-paste code snippets and algorithms, to whole code modules and function libraries, is on the whole far better than to always start from the beginning again (to push it a little).

The downside is, as you point out (or at least infer), that when you add functionality by simply putting together a given set of components (and, yes, I am simplifying this to the extreme), you do not really evolve as a programmer, engineer or whatever.

Just looking at the software engineers around me at work, I know from long experience that a majority of them do not know, and worse - have no interest in learning, anything about the product we are constructing other than the bare minimum they need to produce the document or the piece of code that they are assigned to do.

I am reeling a bit off topic here, but my point is that when the programmers do not need to learn what the code they are constructing will really be used for, and do not need to learn the inner workings of the system since they can just reuse already written and tested components, then most of them just will not bother to do so.

Granted, this is also due to other circumstances, such as that the product we are constructing is incredibly complex, and it would be impossible for one person learn about all of it (and I am just talking about one of the computers in the aircraft - the most complex one of them, but still).

If our software engineers did not have the option to re-use as much code, I am convinced that they would become better at their profession generally, and much greater assets to the project specifically.

Oh, and you may have noticed that I talk about them a lot here. I am of course also included among these software engineers. The exception being that I seem to be a lot more inquisitive and eager to learn new things then the others :-) When faced with a new task, I always take it upon myself to learn as much about it that I can, both in the form of facts and by studying source code (yes, I actually enjoy that too).

Ah - dang, side-tracked again... My excuse is that I have not slept for 32 hours, so my focusing ability is a bit... what was I saying?

If anyone is still reading, my conclusion is that:

Yes, too much reuse of software makes for less knowledgeable software engineers, which makes them markedly less efficient when they actually need to know how the stuff works. Problem analysis is a good example, or even just being able to tell if a suggested design solution is viable. And of course, process improvement is also more difficult to achieve when you do not really know what you are doing :-)

and No, reusing software with care, potentially give you a lot of spare time to consider and plan process improvements.

Answer 5

As pointed out in the accepted answer in another Programmers question, analogies with construction are to be taken with care:

a recommended reading for this is The Software Construction Analogy is Broken

software is often likened to construction. Unfortunately this analogy is flawed and lessons learned from the construction industry are suspect...

What I observed is, good software projects "shift" a lot of repeatability into quality assurance.

When I was a tester in 1,5 year long project, we run test cycles at weekly "checkpoint" releases, about 70 times total through the project. That was... quite repeatable, softly speaking (not much things change in a week). Testing nightly builds has been, naturally, even more repeatable - about 500 times through the project (few entertaining showstopper bugs were too rare to make a difference).

Now, following that "suspect" analogy, tell me a construction company that has built 500 bridges - all with the same team.

• Following it further, tell me a company that has been using mostly the very same bricks and iron at each of their new bridges (here, I refer to the fact that releases we tested had mostly the same bits of code day by day, week by week - "not much things change").
  

---

Master builders are experts recognized for having designed and / or built tens, hundreds, or thousands of things in their area.

Fine, following up your explanation of repeatability quoted above, I can say so what? Back then, our little, not very special group of testers has verified, see above ("about 500") hundreds of things in our area.

As for project developers, they have literally built ("nightly builds") - see, the word is the same, and the meaning is right in this context - hundreds things in their area.

If one wants to continue that "suspect" analogy further up to "thousands of things", these amounts are, again, nothing spectacular in software development when one looks at the right things.

• As an example, yet another of my past projects (again, nothing spectacular, rather regular one), this time in dev role, has been going on for more than 5 years (8 major releases, several dozens minor ones). There were similar weekly checkpoints (5x52~=250 of them), similar nightly releases (5x365~=1800 of them) and similarly, same dev / QA teams working on these. Day by day, week by week, month by month, mostly repetitive stuff (not much change between two nightly builds) - as promised, in the range of thousands times (1800).

Longer lived projects like Windows or Java, or AutoCAD can span 10, 20, 30 years, that easily accounts for repeating as many "thousands" nightly builds and nightly tests as it gets.

---

Concept of repeatability shift to QA becomes even more prominent with Continuous integration...

the practice... of merging all developer working copies with a shared mainline several times a day... CI can be seen as an intensification of practices of periodic integration..

In addition to automated unit tests, organisations using CI typically use a build server to implement continuous processes of applying quality control in general — small pieces of effort, applied frequently. In addition to running the unit and integration tests, such processes run additional static and dynamic tests, measure and profile performance, extract and format documentation from the source code and facilitate manual QA processes. This continuous application of quality control aims to improve the quality of software, and to reduce the time taken to deliver it, by replacing the traditional practice of applying quality control after completing all development. This is very similar to the original idea of integrating more frequently to make integration easier, only applied to QA processes...

Repeatability? it's right there, as much of it as one can think of.

With frequent / continuous QA, things that go wrong quickly get back to developers who are forced to repeat attempts at doing it right until failing tests successfully pass. In a sense, that cycle of repeating until it passes resembles code cata,

an exercise in programming which helps a programmer hone their skills through practice and repetition...

Answer 6

Some of what you say is true: e.g. libraries solve functions not solved by high level languages which solve problems not solved by assembly which solve problems not solved by machine code. When you call System.out.println() in java you are losing sight of how a CPU outputs to a device.

So yes, you're losing something. What you gain is the ability to focus on unsolved problems. Now it may be that you need to immerse yourself in some other aspects of technology ( e.g. how networks function ) to solve a problem. But you don't need to become an expert in reading machine language when all you want to do is build a web page.

The same way, bridge builders are solving a slightly different problem every time ( its a different river ). They don't worry about how to create steel beams of a certain tensile strength, or how to machine bolts to a certain tolerance. They leave that to specialists who have solved that problem.

Look closely, and you'll see our entire society and infrastructure are built on 99% reuse and only 1% true progress. Most new things are just old things with a little extra something added or removed. It's the accumulation of human knowledge. You can write code in a high level language with decent libraries because someone figured out all of the mind-blowingly complicated stuff required to get to this point. It allows you to solve new and interesting problems.

To tie this all together and respond to comments: You don't have to solve problems that have already been solved to develop proficiency. Further, much of what you do will be reinventing the wheel. So in short, the answer is no - you don't need to re-implement the functions of libraries to become proficient. There is plenty of opportunity, some of it rote, some of it creative to hone your craft.

Answer 7

It's all about resources. Years ago, if you developed software projects for large mainframe computers they might be around for 15 years or so with largely a static development environment. The FORTRAN program written to calculate payroll or the COBOL program was perfected over decades because it was constantly being used. There were resources to see how this could be improved. We don't have that sort of slow environment any longer to fine tune and polish the skills for a specific project. But we do take the skills and adapt them to the next project resources permitting. But in the end if becomes a choice of money spent on the new project to do the job, or to do the new job with a large amount of gold-plating. Gold-plating a project means to improve it to the nth degree and add a ton of bells and whistles even if the user didn't specifically ask for it.

The best we can do, is look at the overall design of a new project and see how it can be improved based on the past experience of the team. But it takes a real experience software architect to have a vision about what is actually considered improving the design to improve skills vs simply casing the latest buzzword in development such as Agile, OOP, etc.