Is this limitation of Test Driven Development (and Agile in general) practically relevant?

Question

In Test Driven Development (TDD) you start with a suboptimal solution and then iteratively produce better ones by adding test cases and by refactoring. The steps are supposed to be small, meaning that each new solution will somehow be in the neighborhood of the previous one.

This resembles mathematical local optimization methods like gradient descent or local search. A well-known limitation of such methods is that they do not guarantee to find the global optimum, or even an acceptable local optimum. If your starting point is separated from all acceptable solutions by a large region of bad solutions, it is impossible to get there and the method will fail.

To be more specific: I am thinking of a scenario where you have implemented a number of test cases and then find that the next test case would require a competely different approach. You will have to throw away your previous work and start over again.

This thought can actually be applied to all agile methods that proceed in small steps, not only to TDD. Does this proposed analogy between TDD and local optimization have any serious flaws?

Answer 1

A well-known limitation of such methods is that they do not guarantee to find the global optimum, or even an acceptable local optimum.

To make your comparison more adequate: for some kind of problems, iterative optimization algorithms are very likely to produce good local optima, for some other situations, they can fail.

I am thinking of a scenario where you have implemented a number of test cases and then find that the next test case would require a competely different approach. You will have to throw away your previous work and start over again.

I can imagine a situation where this can happen in reality: when you pick the wrong architecture in a way you need to recreate all your existing tests again from scratch. Lets say you start implementing your first 20 test cases in programming language X on operating system A. Unfortunately, requirement 21 includes that the whole program needs to run on operating system B, where X is not available. Thus, you need to throw away most of your work and reimplement in language Y. (Of course, you would not throw away the code completely, but port it to the new language and system.)

This teaches us, even when using TDD, it is a good idea to do some overall analysis & design beforehand. This, however, is also true for any other kind of approach, so I don't see this as an inherent TDD problem. And, for the majority of real-world programming tasks you can just pick a standard architecture (like programming language X, operating system Y, database system Z on hardware XYZ), and you can be relatively sure that an iterative or agile methodology like TDD won't bring you into a dead end.

Citing Robert Harvey: "You can't grow an architecture from unit tests." Or pdr: "TDD doesn't only help me come to the best final design, it helps me get there in fewer attempts."

So actually what you wrote

If your starting point is separated from all acceptable solutions by a large region of bad solutions it is impossible to get there and the method will fail.

might become true - when you pick a wrong architecture, you are likely to not reach the required solution from there.

On the other hand, when you do some overall planning beforehand and pick the right architecture, using TDD should be like starting an iterative search algorithm in an area where you can expect to reach "global maximum" (or at least a good-enough maximum) in a few cycles.

Answer 2

I don't think TDD has a problem of local maxima. The code you write might, as you have correctly noticed, but that's why refactoring (rewriting code without changing functionality) is in place. Basically, as your tests increase, you can rewrite significant portions of your object model if you need to while keeping the behavior unchanged thanks to the tests. Tests state invariant truths about your system which, therefore, need to be valid both in local and absolute maxima.

If you are interested in problems related to TDD I can mention three different ones that I often think about:

1. The completeness problem: how many tests are necessary to completely describe a system? Is "coding by example cases" a complete way to describe a system?

2. The hardening problem: whatever tests interface to, needs to have an unchangeable interface. Tests represent invariant truths, remember. Unfortunately these truths are not known at all for most of the code we write, at best only for external facing objects.

3. The test damage problem: in order to make assertions testable, we might need to write suboptimal code (less performant, for example). How do we write tests so the code is as good as it can be?

---

Edited to address a comment: here's an example of excaping a local maximum for a "double" function via refactoring

Test 1: when input is 0, return zero

Implementation:

function double(x) {
  return 0; // simplest possible code that passes tests
}

Refactoring: not needed

Test 2: when input is 1, return 2

Implementation:

function double(x) {
  return x==0?0:2; // local maximum
}

Refactoring: not needed

Test 3: when input is 2, return 4

Implementation:

function double(x) {
  return x==0?0:x==2?4:2; // needs refactoring
}

Refactoring:

function double(x) {
  return x*2; // new maximum
}

Answer 3

What you're describing in mathematical terms is what we call painting yourself into a corner. This occurrence is hardly exclusive to TDD. In waterfall you can gather and pour over requirements for months hoping you can see the global max only to get there and realize that there is a better idea just the next hill over.

The difference is in an agile environment you never expected to be perfect at this point so you're more than ready to toss the old idea and move to the new idea.

More specific to TDD there is a technique to keep this from happening to you as you add features under TDD. It's the Transformation Priority Premise. Where TDD has a formal way for you to refactor, this is a formal way to add features.

Answer 4

In his answer, @Sklivvz has convincingly argued that the problem doesn't exist.

I want to argue that it doesn't matter: the fundamental premise (and raison d'être) of iterative methodologies in general and Agile and especially TDD in particular, is that not only the global optimum, but the local optimums as well aren't known. So, in other words: even if that was a problem, there is no way around doing it the iterative way anyway. Assuming that you accept the basic premise.

Answer 5

Can TDD and Agile practices promise to produce an optimal solution? (Or even a "good" solution?)

Not exactly. But, that's not their purpose.

These methods simply provide "safe passage" from one state to another, acknowledging that changes are time consuming, difficult, and risky. And the point of both practices is to ensure that the application and code are both viable and proven to meet requirements more quickly and more regularly.

... [TDD] is opposed to software development that allows software to be added that is not proven to meet requirements ... Kent Beck, who is credited with having developed or 'rediscovered' the technique, stated in 2003 that TDD encourages simple designs and inspires confidence. (Wikipedia)

TDD focuses on ensuring each "chunk" of code satisfies requirements. In particular, it helps ensure that code meets pre-existing requirements, as opposed to letting requirements be driven by poor coding. But, it makes no promise that the implementation is "optimal" in any way.

As for Agile processes:

Working software is the primary measure of progress ... At the end of each iteration, stakeholders and the customer representative review progress and re-evaluate priorities with a view to optimizing the return on investment (Wikipedia)

Agility isn't looking for an optimal solution; just a working solution -- with the intent of optimizing ROI. It promises a working solution sooner rather than later; not an "optimal" one.

But, that OK, because the question is wrong.

Optimums in software development are fuzzy, moving targets. The requirements are usually in flux and riddled with secrets that only emerge, much to your embarrassment, in a conference room full of your boss's bosses. And the "intrinsic goodness" of a solution's architecture and coding is graded by the divided and subjective opinions of your peers and that of your managerial overlord -- none of whom might actually know anything about good software.

In the very least, TDD and Agile practices acknowledge the difficulties and attempt to optimize for two things that are objective and measurable: Working v. Not-Working and Sooner v. Later.

And, even if we have "working" and "sooner" as objective metrics, your ability to optimize for them is primarily contingent on a team's skill and experience.

---

Things that you could construe as efforts produce optimal solutions include things like:

• SOLID principles
• Clean Code
• Design Patterns
• Hiring skilled and experienced people ...

etc..

_{Whether each of those things actually produce optimal solutions would be another great question to ask!}

Answer 6

One thing that nobody's added so far is that the "TDD Development" you are describing is very abstract and unrealistic. It may be like that in a mathematical application where you are optimising an algorithm but that doesn't happen a lot in the business applications most coders work on.

In the real world your tests are basically exercising and validating Business Rules:

For example - If a customer is a 30 years old non-smoker with a wife and two children the premium category is "x" etc.

You are not going to be iteratively changing the premium calculation engine until it is correct for very long - and almost certainly not while the application is live ;).

What you actually have created is a safety net so that when a new calculation method is added for a particular category of customer all of the old rules don't suddenly break and give the wrong answer. The safety net is even more useful if the first step of debugging is to create a test (or series of tests) that reproduces the error prior to writing the code to fix the bug. Then, one year down the track, if someone accidentally re-creates the original bug the unit test breaks before the code is even checked in. Yes, one thing TDD allows is that you can now do a large refactoring and tidy stuff up with confidence but it shouldn't be a massive part of your job.

Answer 7

I don't think it gets in the way. Most teams don't have anyone who is capable of coming up with an optimal solution even if you wrote it up on their whiteboard. TDD/Agile won't get in their way.

Many projects don't require optimal solutions and those that do, the necessary time, energy and focus will be made in this area. Like everything else we tend to build, first, make it work. Then make it fast. You could do this with some sort of prototype if the performance is that important and then rebuild the whole thing with the wisdom gained through many iterations.

I am thinking of a scenario where you have implemented a number of test cases and then find that the next test case would require a competely different approach. You will have to throw away your previous work and start over again.

This could happen, but what is more likely to happen is the fear of changing complex parts of the application. Not having any tests can create a bigger sense of fear in this area. One benefit of TDD and having a suite of tests is that you built this system with the notion that it will need to be changed. When you come up with this monolithic optimized solution from the very beginning, it can be very difficult to change.

Also, put this in the context of your concern of under-optimization, and you can't help but spend time optimizing things you shouldn't have and creating inflexible solutions because you were so hyper-focused on their performance.

Answer 8

It can be deceiving to apply mathematical concept like "local optimum" to software design. Using such terms make software development sound much more quantifiable and scientific than it really is. Even if "optimum" existed for code, we have no way of measuring it and therefore no way of knowing if we have reached it.

The agile movement was really a reaction against the belief that software development could be planned and predicted with mathematical methods. For better or worse, software development is more like a craft than a science.