I would say the "design dead-end" phenomena is orthogonal to agile methods. What I mean is that it is possible to do waterfal, spend a lot of time upfront on a (bad) design. Then spend a lot of time implementing it only to find yourself at a dead end.
If anything, agile methods should help you discover earlier that you made bad design choices. The reason for this is that your backlog should have the highest customer value items done first and you should focus on delivering useful increments of the software. If your design allows you to deliver high value and usefulness it's already good for something :-) In contrast, you could have a bad design in a waterfall-ish situation where you may not find out for many years that this design can't deliver any value and any usefulness - all you have is the illusion of it being a good design. As they say, the proof is in the pudding.
The flip side is that even in agile methods it is important to have a viable vision for the system's design that drives decisions from iteration to iteration. I think Ken Schwabber said something like if you have a team of terrible developers they will produce bad software consistently iteration by iteration. Agile simply means don't spend a lot of time upfront because you are limited in what you can learn or imagine before you start implementing (and the requirements also change). However, there are some situations where you have to do upfront work (e.g. research) and then you gotta do that.
How do you avoid dead-ends?
I would say mostly by anticipating future requirements. This is something you get with experience and familiarity with similar projects/products. This anticipation is partly what helps you put a good design in place because you ask yourself a lot of "what if" questions about your current system. To me this is the critical component. Techniques like OO are simply helping you when you already know what you're doing.
What do you do if you have a dead-end?
A "dead-end" is no different than any other technical block you will hit during the development of anything that is novel. The first thing to realize is that there are really no true "dead-ends" that force you to completely backtrack. At the very least your learning up to this point is what enables you to go forward so the effort has not been wasted. When you hit a dead-end you have a problem. The problem is what needs to change in order to meet some new (or old) requirement and how to optimize making this change. All you have to do now is solve this problem. Be thankful that this is software and not, e.g. an airplane design, because change is much easier. Identify the issues, fix them == refactor == software engineering. Sometimes a lot of work is involved...
A few famous big requirement changes from the OS world that come to my mind:
- Apple moving from 68K to PowerPC.
- Apple moving from PowerPC to Intel.
- Windows needs to run on ARM.
- Adding support for symmetric multi-processing to a single processor OS.
Whichever way you look at these they are a lot of work. The original design almost certainly did not take into account the possibility of this happening (i.e. portaility was not a big requirement). Whether the design was OO or not is probably not a huge factor either. In a good design the platform specific portions would be somewhat isolated and the work would be easier.
