I work on a fairly large software system and over the years it has accumulated a lot of entropy. There is plenty of scope for refactoring but there is always pressures to build the next features upon what's already there. This adds to more entropy, because design choices for implementing new features are typically made by first accepting what's there and 'working-around' some earlier weaker design that is otherwise ripe for refactoring.

What are some ways to manage this kind of complexity and build functionality without substantially weakening the structure of the system further. I know this is a very broad question and the approaches/solutions depend on the particular software system at hand, but I am still hoping there are some generic ways to manage the problem I am highlighting.

  • 2
    Perhaps relevant is this question about how to convince management to deal with technical debt. I'm not sure that it's an exact duplicate, but it may have some helpful material. – Thomas Owens Mar 10 '13 at 22:13
  • Would you or someone responding to this question also explain what you mean by the term 'entropy' in this context. Are you using this word in a qualitative sense or is there actually something measurable? thank you – Mark Rovetta Mar 10 '13 at 23:39
  • As a qualitative measure, I sense it as how difficult/time consuming it has been for me to modify code/extend functionality that was implemented by another programmer. Also how often I've unwittingly broke another part of the system because I violated an assumption that wasn't documented and was probably made to quickly fix an issue under time pressures in the past. Also reuse becomes harder. That's something that kinda does what I need but makes assumptions about the sequence and the context in which that module is called, so I need to set things up just right for that module to do its thing. – Ranjit Iyer Mar 11 '13 at 0:29
  • 1
    @MarkRovetta Software entropy is similar to software rot and is related to the idea of entropy from thermodynamics. As software systems grow in terms of scope and functionality, they become more complex and more difficult to maintain. Investing effort can reduce the increase in entropy. – Thomas Owens Mar 11 '13 at 10:07
  • 1
    I found two other questions that may also be relevant: One is about maintaining sanity when refactoring poor code and the other is about what do to after inheriting bad code. If none of these three questions address your concerns, consider revising your question to be more specific about your situation. – Thomas Owens Mar 11 '13 at 11:52

You need apply Separation of Concerns principle in order to separate the system in modules which only communicate between each other other via very well defined interfaces. This greatly reduces complexity due to many reasons:

  • Some can work on a part of the project without understanding the other parts;
  • You can easily improve the system by adding another module;
  • Loose coupling - the modules are only connected by interfaces;
  • High cohesion - the data from a module is used in almost all the parts of that module.
| improve this answer | |

Ah yes, this problem.

The first thing you have to do is to get management on board with the idea that the software needs to be made evolvable (my term). The idea here is that instead of a system of fixed design and changing requirements, you have to get them on board with the idea of modifying the design as the requirements change. Otherwise you invariably end with monsterous systems that are impossible to maintain.

The second thing (once you have management on board) is that you have to get a commitment from programmers to separate contract from implementation. Then you have to be willing to write unit tests against contracts, not against implementation.

Now if this is a large-scale system (and it sounds like one) the next stage is actually to build a new framework. New code goes on the new framework and eventually the old framework will be retired. You should do this in a way that allows you to preserver old contracts (with external apps) in the new framework for as long as you can.

From there, the solution is what I call "refactor with a chainsaw" and remove well-defined blocks of functionality from the old framework and move it onto the new framework. On a large scale system this can take years to move over everything.

Now if you have unit tests on contracts, then you can refactor your own implementations as you go without worrying as much about breaking things.

| improve this answer | |

You can't. As you already acknowledge, every new feature built on the weak foundation makes that feature weaker, perpetuating the cycle.

What you need to do is spend the time to refactor things to at least not require hacks for new functionality. Maybe that means a heart-to-heart with management about tech debt and/or maintenance costs. Maybe it means having high estimates to make sure you get the time to do it right. Maybe it means having the pride at your craft to take the time to do it right, even if it doesn't get done 'on time'.

But you're the one writing the code - so you're the one who is going to have to fix it. How you do that will vary from place to place.

| improve this answer | |

One reason that systems get into this state is that there has not been anyone guiding the changes. Many successful long-term projects have associated steering committees and roadmaps to guide development. These are especially helpful when there are multiple development teams and stakeholders involved.

My suggestion is that your team develop a roadmap to refactor the system. First, define a goal. In your case, define the ideal architecture for the system. Remember, it doesn't have to be perfect, just workable and better than what you are working with. Take into account software, development tools, hardware platforms, major new features that may be added to the system. Do a cost analysis. If you have the desired architecture, what are the tangible benefits of performing the refactoring? (Benefits would include reduced test time, more reliable software products, more predictable development cycles, etc.) If you can show enough benefits to performing refactoring, you can get management support.

Once you have a roadmap and management support, Develop a series of steps that you need to take to get to this desired state. It may be a good idea to put together a steering committee that maintains the roadmap, trains and educates the development teams on the roadmap, and periodically reviews changes for architectural integrity.

Roadmaps are really useful, and perhaps the 13th question on the Joel Test should be "Do you have a roadmap?"

Here is a video of the first hour of a recent Red Hat Linux roadmap session.

| improve this answer | |

I must say I don't really like the unit testing, and I still see it as a partial patch which doesn't fix good design, technical debt or any of the multi-threading issues. But, I think that the unit testing might be the only way to fix the issue.

After things get really bad, only the unit tests can help to get stuff under control.

Or a rewrite with skilled developers, but that will loose features, some of which might be bugs that customers perceive as features.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.