How is Agile model more flexible than the Waterfall model?

Question

I know one of the differences between Agile and Waterfall is to do with more customer and user feedback, but I'm trying to wrap my head around what makes the Agile/XP model more adaptable to change.

For example, say I'm developing a system that lets users write their own notes. It would involve designing, developing and testing the program before deploying it to production, which both models do (aside from the Agile model allowing you to do it with multiple iterations).

But, in the middle of development, someone changes their mind about what to add to the program (for example, add a feature that lets users add not only text, but also videos and images to their notes), then Agile would allow developers to go another cycle to implement that feature.

But Waterfall also lets you do the same thing (go back to the design and then write the code for it), except it claims to be "slower" and "more difficult to go back".

Am I missing something about both models?

Answer 1

While it's been massively subverted over the years, the idea behind agile isn't that you deliver the same set of features faster. It's that you get things out to your users for feedback quicker and earlier, so you're more likely to be building the right thing.

Answer 2

I think experts generally accept that "Waterfall" is a stereotype of certain ineffectual development practices.

From the 60s into the 90s, I dare say the vast majority of software developments - at least for business data processing - were not big-bang, one-shot projects conforming to "Waterfall", but incremental developments performed and readapted over many years by long-serving, permanent in-house staff. Let's call this "original agile", although I'm assigning the term retrospectively.

Amongst all the failures and uncertainties of this original-agile approach, and amongst an always-increasing software complexity, there was a growing tendency towards the stereotypical Waterfall practices.

The intention of Waterfall practices was to increase control and reduce risk, but in practice it failed to do those things. Instead its overheads just added to the cost, and its increased rigidities reduced the quality.

The intention of Waterfall was also for organisations to start treating software development like mechanised manufacturing, where the employer owns the machinery, the processes, the tools, and the know-how, and applies workers to perform the routine activities within an overall division of labour. This was as a fightback against the new kind of powerful craft workforce emerging in software development, just as the power of old craft workforces was being successfully destroyed by advancing mechanisation.

It's also worth noting that original-agile was completely inconsistent with the outsourcing and contracting business models that increased especially in the 90s, because with no limit on price nor with criteria for delivery, unscrupulous capitalist contractors would naturally just take the client for a ride and deliver nothing - or certainly charge twice as much as directly-employed labour would cost for the same delivery. So this was another reason to move towards Waterfall, because supposedly you could fix a price and hold a contractor legally liable for delivery to a spec.

The Agile Manifesto was a fightback against those currents. It reasserted a number of things, including the need for the skilled craft workforce over the use of drones following routine processes furnished by the organisation.

And it also invited contractor-clients to accept the reality of how development is done and always was done, and therefore accept the need to furnish money speculatively with no guarantee of quality or success. Control relies on reputations and a sense of professional purpose, and the power of the contractor-client to "sack" the contractor as the final disciplinary measure - which is much the kind of control employers have on directly-hired staff. I know this isn't how Agile contracting is generally sold, but is the reality of how it works in practice.

Meanwhile the so-called "Agile-Industrial Complex" nowadays is basically just another iteration of the forces behind Waterfall. Written documentation may be lighter, but the fundamental point of putting the workforce to work on routine tasks within a division of labour remains.

The equivalent of the old heavy documentation is now the heavy training and credentialism of "scrum masters" and non-technical "product owners" and various kinds of "daily ceremonies" and that sort of thing - the point is for the employer to own something, in this case an organisational structure and a set of routine practices (rather than the paperwork and routine practices of yesteryear's Waterfall), which supposedly allows it to hire drones and, if necessary, simply sack the incumbents and place new faces in all the same positions, and the work will carry on as before.

There is really only one way development is actually done, and ever has been done, and it is broadly consistent with "agile" on the supposed Agile-Waterfall axis. The essential meaning of what Agile is, is highly obfuscated by writings on the topic which come from the perspective of the Agile-Industrial Complex, but the crucial point is that basically development today, when done effectively, is being done on much the same terms as it was done in the 1960s.

Answer 3

But Waterfall also lets you do the same thing (go back to the design and then write the code for it)

Not really. Agile is a reaction to big project management. So the waterfall they are talking about "go back to design" means throwing away years of work and invested money. Essentially killing the project.

For example to get to the stage where you start programing, your notes app would have had to have gone through:

• market research
• feature design
• architecture
• cost analysis
• breaking up into components
• UML diagrams for each component
• estimate and budget

and god knows how many more steps, all involving lots of meetings and sign offs at each stage and paying people money to complete that stage.

someone changes their mind about what to add to the program

Who? All the features have gone through all these stages with sign offs and committees at each one. You want to change a feature? have you done the market research? what about budget? it will need to goto the architects of course..

The point of the big waterfall process is to make sure you have dotted all the i's and crossed all the t's, you can't make impulsive changes and if you do its expensive.

Agile is saying "that doesn't work for software"

Answer 4

I think it's important to understand that the idea of the waterfall process, as most people understand it, was an error. I don't know exactly what went wrong but if you read the original waterfall paper by Winston Royce, he describes the outline of the process which came to be known as waterfall. Right after that, there's this line:

I believe in this concept, but the implementation described above is risky and invites failure.

Emphasis is mine. Somehow, this key observation was lost, and several decades were spent 'inviting failure' before it was generally accepted that the approach wasn't working. And to be sure, failure showed up again and again. I really encourage everyone interested in software development processes to read the original 'waterfall' paper linked above. If you don't stop reading after the first page-and-a-half, you'll find that Royce advocates a process much more like Agile than the waterfall process (as it came to be understood.)

In a nutshell, I think the answer to the question you are asking is that, in 'classic' waterfall, you spent a lot of time getting all the requirements and design done before you start developing things. This is simply inefficient. That is, it takes a longer time to get the same result you get with 'agile' methods. Often there's not enough time for the process to actually produce a good (or even a working) result.

There are a couple of reasons for this. First off, what are the developers doing while the requirements and design are being completed? Hopefully you have some other work for them to do for those months. The second big issue is that the end product almost never looks like what was first envisioned. My experience is that you can't really get good requirements from people until they see the software and start telling you what is wrong about it. Often the requirements you get prior to that are fundamentally flawed. They might not be feasible, solve any real problem, or fail to consider many factors that are key to the success of the product. All that time spent writing down those wrong ideas and designing around them is simply wasted. You are far more likely to get a working result quickly if you start creating software that can be put in front of the people who will be using it sooner.

Answer 5

The difference is what "in the middle of development" means.

The Manifesto for Agile Software Development was a response to heavyweight, documentation-driven, command-and-control processes. These weren't necessarily "waterfall" processes - one example could be Barry Boehm's spiral model.

The fundamental problem that they saw was the inefficiencies of the documentation. Just because everyone agreed on the requirements specification, the architectural specification, and the design specification doesn't mean that they accurately reflected what the customer needs or wants. This holds true even if the customer is involved in creating and reviewing the documentation. The best way for the team to understand if what they are doing is correct is by having someone interact with the working software and give feedback.

By working in smaller increments and building smaller portions of the system, the team can get feedback earlier. Other practices associated with Agile methods also help the team focus on valuable work. These practices include executable specifications (and TDD, ATDD, and BDD), continuous documentation, just-in-time decision making (a practice from Lean), and favoring collaboration over gates and approvals.

For a team working in heavy, document-driven processes, a change made 4 or 5 weeks into the process to a requirement could impact a series of architectural, design, and implementation decisions. However, a team working in an Agile manner would have deferred more of those decisions and reduced the impact of the change.

Agile is not about speed. It looks like the team is working faster because they are delivering faster. Speed is also a byproduct of adopting to change and reducing the rework associated with changes.

Answer 6

This might be a subtle thing to point out, but I believe it lies at the heart of understanding the difference between agile and waterfall:

But, in the middle of development, someone changes their mind about what to add to the program (for example, add a feature that lets users add not only text, but also videos and images to their notes)

In agile, someone didn't change their mind. They just specifically hadn't made up their mind yet, because they hadn't yet gotten to the note-content-building stage.
They hadn't yet gone through the process of locking down the requirements, doing the technical analysis, and preparing to implement the note content certain way, before then "changing their mind". Even if they had an initial idea and now have a different idea, they don't need to redo a whole lot of documentation on the initial intentions, because they didn't lock the first idea down ahead of time.

They simply deferred the decision on what the note's content should be until the time when they actually needed to build it. The reason they deferred it is because their future selves would have more knowledge than their past selves (i.e. early user feedback based on an early release), and therefore their future selves would be more likely to make the right call.

But Waterfall also lets you do the same thing (go back to the design and then write the code for it)

Waterfall has you design the whole thing before you've written the code, let alone deployed it and asked for user feedback. In a waterfall process, someone would have definitely already locked in what content a note should contain, done the technical analysis, and prepared for the codebase to be built a certain way.

That's not to say that waterfall processes don't allow for building change-friendly codebases, but the end result here is that if you go back to the drawing board on what a note should contain, then you will have wasted effort on how you designed this the first time (which you're now overwriting with new requirements).

By not deferring the analyses (requirements and technical), you run the risk of prematurely locking something down and then needing to revisit it when you have future knowledge of things you weren't aware of at the original time.

then Agile would allow developers to go another cycle to implement that feature.

I just want to point out here that if an agile process had already delivered the first version of the note content, then the cost of going back and revisiting note content is the same regardless of whether you're working with agile or waterfall.

We can argue that agile generally puts more pressure on having a change-friendly codebase and whether or not that distinction matters for the above statement. For an identically written codebase regardless of using agile or waterfall, the above statement is correct.

The only time agile prevents you from wasting time and effort is when you end up changing your mind after the dev work has already started (and/or after a version was already released) and before you got to the part that you've now changed your mind on.

If it helps, you can define agile as a series of consecutive waterfall processes, but only if those waterfall processes do not attempt to guess at what future iterations will need to contain. The expectation of being able to predict the future is what sets waterfall apart from agile.

Answer 7

In addition, to what @Philip is saying:

Agile recommends rolling out one feature after another, all the way to users/consumers, whereas Waterfall recommends completing initial phases for all the features and then rolling that out for testing.

But, in the middle of development, someone changes their mind about what to add to the program (for example, add a feature that lets users add not only text, but also videos and images to their notes), then Agile would allow developers to go another cycle to implement that feature.

That "someone" is often an end user/consumer, so cycles happen faster with Agile because the consumers are involved more quickly as a matter of priority.

Agile makes the case that complete program design cannot always be known without end user/consumer input, so better to roll out feature by feature to the end user/consumers than to complete program design first.  (To the extreme, Agile might say that consumers don't really know what they want, so prototype something and let them try it out, rather than trying to tease all requirements out of them up front.)

But Waterfall also lets you do the same thing (go back to the design and then write the code for it), except it claims to be "slower" and "more difficult to go back".

Sure, it just recommends completing steps like "Program Design" for the whole project before moving to the next, like "Coding".  By its nature, this will put more complete software into end user/consumer's hands, yet later — whereas Agile recommends putting less complete software into end user/consumer's hands sooner.

There's also the nature that the initial stages of Waterfall don't have good or as formal support compared to the formality offered by programming languages.  Yes, there's tools to help with diagramming requirements, analysis, and design, but they are not as formal as programming languages with compilers.  (So, in the extreme, Agile recommends that design is captured in code rather than less formal design documents.)  (Compile-able & executable design documents would be a plus here.)

Answer 8

Fast Failure

Agile (at least the flavours that I know best, XP and Scrum) have the idea of failing fast. If you are doing the wrong thing, you should stop doing it as soon as possible. I was once brought in to close down a project with as little pain as possible. Management had identified a business problem, specified a solution, designed to the spec, implemented it, tested it, shown it to the users (sales team), who said that it sucked and refused to use it. The solution was wrong: instead of trying to make a bad manual process more efficient, they now decided to stop doing whatever it was they did. Agile would probably have failed too, but it would have failed earlier, cheaper, and without demoralizing the team.

Projects that suit waterfall

There are some projects that suit waterfall. If you are planning a space mission, and you can predict which celestial objects will be where at each future time, and you absolutely cannot fix things after launch, you had better write a spec. If you are doing a technology refresh on existing software, e.g. rewriting a legacy medical records system with new technology, waterfall is likely to help. If the FDA needs to approve your system, they will want to read doco, waterfall will help.

If you don't understand where you are going, be Agile

Imagine that you are building a completely novel application. Your users have never seen this type of application before, so they can't tell you what they want. Once they see it, though, they will think of new ways to use it, which never occurred to you. I have met developers who use waterfall as an excuse: sorry, that's not in the spec. The alternative is to get them working code, albeit incomplete (a former manager of mine used to talk of a "vertical slice" through the system), and let them figure out what a good system will do.

You might find it interesting to read about Bertha Benz's cross country automobile ride in 1888.

Although the ostensible purpose of the trip was to visit her mother, Bertha Benz had other motives – to prove to her husband, who had failed to adequately consider marketing his invention, that the automobile in which they both had heavily invested would become a financial success once it was shown to be useful to the general public; and to give her husband the confidence that his constructions had a future.

While she achieved this, her trip also revealed design flaws, which she was able to solve. Arguably she was the first motorist ("user"), so she tested the vehicle in ways that the designer hadn't envisaged.

Answer 9

When I first read the question title it seemed so simply I felt like voting to close it.

Sure, with waterfall design you can go back and change it. But it's so much extra work, so demoralizing taking apart code that's been tested extensively, so annoying that the client didn't say anything about a feature they just "took for granted", etc.

Every sphere of human activity has its humbug: the things people have in mind but don't speak explicitly on, preferring to go on and on about some other more presentable aspect - usually an aspect that requires acceptance of that aspect too shabby to speak openly about. Lab technicians in a university go on and on about "safety" when additional work is added to their labs. Teachers talk about "educational quality" - and often go on strike - when additional tests are introduced. Junior doctors in hospitals complain of the dangerous risks to "patient care" due to tiredness from their high working hours. We all know that the major component of the final alleviation to such complaints will be either less work or more money.

Software development is no different. "Flexibility" is not just a physical thing - being able to change a design easily. It is also a euphemism for not getting locked into a difficult or low-margin commercial relationship with a client. (An agreed set of deliverables can be justifiably linked to a corresponding set of stage payments - vital for cash flow.)

To some degree it's also a euphemism for people who like a more varied and exciting workplace with socio-professional relations in a way they find more agreeable - more group discussions, more creativity and, for some, more competitive drive with their co-workers so their abilities are noticed.

Answer 10

[Day 1] Objective: From London, take us to Edinburgh.

Doing this the "Waterfall" way, you'd hop on the motorway and drive to Edinburgh.
You'd ignore the weather, turn off the SatNav, sit in endless traffic jams, not look for alternatives.
You'd just drive to Edinburgh.

Doing this the "Agile" way, you'd set off on a smaller road, and only plan as far as the next town. When you get there, you'd stop, review your progress, see what's working and what isn't and plan the next leg of the journey, to the next town. You'd "rinse and repeat" this at every town between start and finish of your journey.

Now, OK, that makes Agile sound really bad ... except that your journey might well finish up in Bristol, which where you actually need to be.
The original objective - Edinburgh - has been repeatedly re-evaluated along the way, found to be "lacking" in some way and decisions taken - collectively - that Bristol is the better place to go. So you wind up where you need to be, not where you was originally wanted to be.

Answer 11

The basic concepts that make Agile more flexible, if done correctly, are these:

• Thinking more in terms of MVPs (minimum viable products) for each feature. In practice, this goes hand-in-hand with the basic tenet that the individual stories should if at all possible be formulated in a minimum size. Or in other terms, any story, when finished, should be possible to be rolled out to production immediately. Optimally it should already increase the value of the product for the user; but as a baseline it at least needs to be possible to disable it with a feature toggle or something like that. But you never want long-term unmerged, molding, development code. I recall older projects where whole subsystems were developed as one piece over months or years, and then there was a big bang deployment (maybe after the original developers were gone...).
• Stories are not fully refined long before they are ever worked on, but ideally relatively shortly before being put to active work. This is reflected in some story status schemes containing something like "backlog", "accepted for refinement", "accepted for implementation", "in implementation" etc. - i.e. the process of ideation, over (business) decision, over refinement, leading up to implementation, is part of the everyday process. In contrast, in the good old days of fully planned-out waterfall projects, the whole project was sometimes planned, in detail, long before the first line of code was written. This made it very hard to change things, or at least very costly (as the original cost was already sunk).
• If doing sprint-based development, each sprint backlog should, when planned and commited to, be relatively static and unchanging, for the duration of the sprint. This is often misunderstood by stakeholders; they believe that Agile means they can switch stuff around daily. Nothing could be further from the truth. With agile processes, roles like a Scrum Master (or "Agile Master" if you want to avoid mentioning Scrum) were invented which take care of the process. In old projects, this was often the purview of the project lead, who often had very different priorities on top.
• Another core tenet of Agile is that the developers themselves have a strong voice in the fine-planning of the stories, in committing to short-term timeframes (i.e., the sprint) and so on. In the distant past, waterfall projects were often pre-planned by a completely different team, which maybe didn't even have experience developing the actual software...
• Also, especially if doing sprint-based cycles, the sprint change is always a good point in time to do reflection about what went well and what didn't. If you figure out that a story or feature just didn't work out, worst case you lose the time spent within that particular sprint. Yes, you can do the same in waterfall mode, but it's not something that comes natural.

No discussion about Agile should be without mentioning the Agile Manifesto. It contains little arguments for "why" it says what it says, and many of its topics are a bit more abstract, but much of it should be relatively self-explanatory.