What do you do if you reach a design dead-end in evolutionary methods like Agile or XP?

Question

As I was reading Martin Fowler's famous blog post Is Design Dead?, one of the striking impressions I got is that given the fact that in Agile Methodology and Extreme Programming, the design as well as programming is evolutionary, there are always points where things need to get refactored.

It may be possible that when a programmer's level is good, and they understand design implications and don't make critical mistakes, the code continues to evolve. However, in a normal context, what is the ground reality in this context?

In a normal day given some significant development goes into product, and when critical change occurs in requirement isn't it a constraint that how much ever we wish, fundamental design aspects cannot be modified? (without throwing away major part of the code). Is it not quite likely that one reaches dead-end on any further possible improvement on design and requirements?

I am not advocating any non-Agile practice here, but I want to know from people who practice agile or iterative or evolutionary development methods, as for their real experiences.

Have you ever reached such dead-ends? How have you managed to avoid it or escaped it? Or are there measures to ensure that design remains clean and flexible as it evolves?

Answer 1

I just read the article link you posted, I have to say Fowler has made some very good points and a lot of things he said, I've been advocating with our team for years.

IMO, if you do any decent design, you should not get into what would be considered a dead-end situation. I've always viewed software as made up of building blocks. I still believe in some up-front design, but the main goal is not to design the entire product, but to provide overall architecture/direction so your team can visualize a common picture that we are all working towards. If you have a bunch of cube and triangle pieces, it's helpful to sketch out how a castle would be put together before you simply start slapping pieces together.

Since I come from OO land, to me each block is a class and the surface area of that block is the public interface (what is visible by external or deriving classes). If you follow good SOLID principles, you will make sure that each block as extremely simply and has an intuitive public interface. Going back to my analogy, you want to make sure you code only creates simple shapes. Whenever you create classes, that are too complex (many functions, many variables), you creating shapes which are hard to reuse when requirements change.

I agree with Fowler in that the biggest risk/challenge for evolutionary design is that you leave design decisions to coding time, and you expect each individual developer to make those decisions. This is where the system can break down if you don't have proper feedback mechanisms in place. Whenever a new feature is asked for, it is extremely tempting to simply find the function that needs to be extended, put some kind of conditional inside it and just add a whole bunch of code right inside that function. And sometimes, this might be all that's needed, but this is also (IMO) the single most common practice that leads to dead-end components. This has nothing to do with evolutionary design. This is what's called "no design".

As long as you take the time to step back and say, wait a minute, this class already has 15 member variables, let me extract 6 of these and put into their own self-contained class, your software will be made up of very light-weight, flexible and reusable building blocks. Sure if PMs come along and change half the products requirements on you, you may have to take some of your blocks out, put them back on the shelf and draw up some new ones (just like when building a castle, you may not use all your cylinders). But at that point, that's just part of doing business. Requirements changed and by keeping your code flexible and modular, you should be able to change your product to align with your new business direction.

I believe this evolutionary approach to design works with every level of engineer's skill. Personally, I've done software for a very long time and before our team moved to agile methodology, I was responsible for shipping several major components from my dev PC almost directly to the customer with barely any QA. At the same time those components have always remained flexible and maintainable.

I'm only trying to say that I'd consider myself relatively decent at designing software. At the same time, if you asked me to write up a 100-page design document, give it to a coder and expect it to work, I probably couldn't design myself out of a paper bag. When starting work, I would sometimes sketch out few UML-like (very simplified, not full language) diagrams, but as I start coding, I would refactor on as-needed basis and my final code would never look like what I originally drew. Even if I spend a month or two thinking about every little detail, I can't imaging someone else being able to take my diagrams and come up with solid piece of software without modifying the design as they are coding.

On the other end of the spectrum, currently in my team (now agile and I fully support that) we have a couple of guys who joined us from embedded land where they have only done C for the last 15 years. I obviously helped with some initial planning and laying out classes but I also made sure to follow up with regular code reviews and brainstorming sessions where we discuss applications of SOLID and design principles. They did produce some spaghetti code that made me cringe a little, but with just a slight nudge from me, they started refactoring what was already produced and the funny part is that one of them came back to me few days later and says, I hate to say it but after moving that code out, this looks so much more readable and understandable. Dead-end averted. Point I'm trying to make is that even someone who is completely new to OO can produce somewhat decent code, as long as he has a mentor with more experience, to remind him that "evolutionary design" is not same thing as "no design". And even some of his "more complex" classes aren't that scary because each class doesn't have that much responsibility (i.e. not that much code), so worst comes to worse, if that one class "dead-ends", we chuck it and write a replacement class that has same public interface (so far I never saw a need for this contingency in anything we wrote and I've been doing code reviews twice a week).

As a final note, I'm also a firm believer in design documents (at least for the business conditions of my current team) but the primary goal for our design docs is Organizational Memory, so actual documents are written after the code is produced and refactored. Before coding, we generally have a quick (sometimes not so quick) design phase where we sketch out classes on napkins/mspaint/visio and I always remind people that this phase produces a path to follow, not a blueprint and as they start coding, anything that doesn't make sense should be changed. Even with these reminders, newer guys tend to try to back fit code into original design no matter how unnatural it feels even to them. This usually surfaces in code reviews.

Dang, I wrote a lot. Sorry about that.

Answer 2

I would say the "design dead-end" phenomena is orthogonal to agile methods. What I mean is that it is possible to do waterfall, spend a lot of time upfront on a (bad) design. Then spend a lot of time implementing it only to find yourself at a dead end.

If anything, agile methods should help you discover earlier that you made bad design choices. The reason for this is that your backlog should have the highest customer value items done first and you should focus on delivering useful increments of the software. If your design allows you to deliver high value and usefulness it's already good for something :-) In contrast, you could have a bad design in a waterfall-ish situation where you may not find out for many years that this design can't deliver any value and any usefulness - all you have is the illusion of it being a good design. As they say, the proof is in the pudding.

The flip side is that even in agile methods it is important to have a viable vision for the system's design that drives decisions from iteration to iteration. I think Ken Schwabber said something like if you have a team of terrible developers they will produce bad software consistently iteration by iteration. Agile simply means don't spend a lot of time upfront because you are limited in what you can learn or imagine before you start implementing (and the requirements also change). However, there are some situations where you have to do upfront work (e.g. research) and then you gotta do that.

How do you avoid dead-ends?

I would say mostly by anticipating future requirements. This is something you get with experience and familiarity with similar projects/products. This anticipation is partly what helps you put a good design in place because you ask yourself a lot of "what if" questions about your current system. To me this is the critical component. Techniques like OO are simply helping you when you already know what you're doing.

What do you do if you have a dead-end?

A "dead-end" is no different than any other technical block you will hit during the development of anything that is novel. The first thing to realize is that there are really no true "dead-ends" that force you to completely backtrack. At the very least your learning up to this point is what enables you to go forward so the effort has not been wasted. When you hit a dead-end you have a problem. The problem is what needs to change in order to meet some new (or old) requirement and how to optimize making this change. All you have to do now is solve this problem. Be thankful that this is software and not, e.g. an airplane design, because change is much easier. Identify the issues, fix them == refactor == software engineering. Sometimes a lot of work is involved...

If you use Scrum this change should naturally be driven from user stories (what does the user get from this change?). The process would start from one story which can't easily be accomodated by the current design (oops) and a discussion would happen with the product owner about how to break this story down. You keep applying agile principles through this change.

A few famous big requirement changes from the OS world that come to my mind:

• Apple moving from 68K to PowerPC.
• Apple moving from PowerPC to Intel.
• Windows needs to run on ARM.
• Adding support for symmetric multi-processing to a single processor OS.

Whichever way you look at these they are a lot of work. The original design almost certainly did not take into account the possibility of this happening (i.e. portaility was not a big requirement). Whether the design was OO or not is probably not a huge factor either. In a good design the platform specific portions would be somewhat isolated and the work would be easier.

Answer 3

I refactor my projects permanently and also use UML class diagrams. I mean that I create one or more class diagrams by packages. Each diagram is saved at the root of the package. Each UML classifier has an own Id which is mapped to the related Java Id. It means that when I open my diagram it is automatically updated to the latest code refactoring changes. I can also directly change my class diagrams at graphical level and all my project is immediately refactored. It works pretty well but it will never replace human. My UML class diagram is also only a graphical view of my code. It is very important not to be mix code and model like EMF eclipse is doing because as soon as refactoring is done then model information is also lost. I never use Model Driven Development code generator because this is useless. I don't like model drives my code but my code drives my UML class diagrams.

Having said that having over 100 class diagrams representing all details of my project structure and full of notes everywhere is really helpful. I only create class diagrams for projects because usually developers don't have time to learn or use other diagrams. Class diagrams are also so good because automatically updated. Class diagrams can be created after the code by just reversing a package and adding notes. It is fast and always accurate and 100% iterative.

Please don't make confusing between model driven development which is a model generating code and usually using UML as graphical presentation with UML class diagrams updated from the code. Only UML synchronized code has a real value for me if multiple iterations.

Sorry to be so long but I think we should give a second chance to UML class diagrams if only used as a graphical view of our project. It means that UML covers the full project and have a single model composed by large class diagrams representing the full project. It would be ridiculous to have hundred of small views and a model for each view within a project having hundred of views :-)

Answer 4

I have reached dead end my code and others code due to bad design, change in direction, etc. I've also seen many others run into this problem. The big mistake (at least it seems like a mistake to me) is the immediate desire to throw away working code and reimplement everything from the ground up.

I approached each case in the same way which seemed to work well:

• identify why the current design isn't working
• come up with a new design and a transition plan
• tag the code that will not be used as deprecated
• implement only what I need for the new design in order to get the immediate need fulfilled
• remove the code the new code makes obsolete

Costs:

• more complexity due to 2 implementations in the code base at the same time
• more total cost per feature/bug fix than doing a full redesign/reimplementation assuming good use cases and tests

Benefits:

• at least an order of magnitude less risk because you still have the old working design/code
• faster to market for any 1 to n-1 features that don't require a full redesign
• if the product/code ends up dieing before you've completed the full redesign you will have saved development time difference
• iterative approach and all the benefits therein

Answer 5

About a month or two ago, our current project got a bit stuck due to some bad design decisions (and lack of much design in one place), with the SCRUM development style.

Our solution (and what I believe is the standard one for SCRUM) was to devote an entire sprint (~2 weeks) to nothing but refactoring. No new functionality was added during this time, but we were able to think about the current codebase, and design a much better system for what we were doing.

We are now past that hurdle, and have been adding in new features again.

Answer 6

The key to limiting the cost of design changes is keeping the code as DRY as possible. This will drive most of the application code to a very high level, where most of the code directly expresses the intention, and relatively little specifies mechanism. If you do this, then design decisions will have the smallest possible expression in the code, and design changes will have the least possible cost.

Answer 7

The key to avoiding design dead ends is to recognize as early as possible when your design needs to change, and change it then. The biggest problems come not by continually evolving your design, but by refusing to evolve your design until it is a huge problem.

As an example, Netflix has a profile feature, where different family members can bill to the same plan, but have separate queues. A few years ago, they announced they would have to cancel that feature, because only some 10% of their users used it, but due to the hacked on implementation, it was eating up an inordinate amount of maintenance work. After an uproar, they bit the bullet and did an expensive redesign in order to keep those customers.

I'm sure there were some engineers who recognized a suboptimal design when they first added that feature. If they had changed it back then, it wouldn't have been nearly as big of a deal.

Answer 8

Wasn't it Fred Brooks who said something like "Plan on throwing the first one away"? Don't feel too glum about it, dead-end designs also pop up in projects that try to do all the design up-front as well. Redesigns happen in all types of development, whether because it was an unworkable design from the start (that last 20% that gets glossed over- "the devil is in the details") or because a customer changes their focus. There is no real need for alarm bells, don't be too worried.