Are there advantages to agile practices other than having a working build between sprints?

Question

I recently became interested in agile practices in software development and I since have seen a lot of articles point out that these practices allow for reduced overall costs.

The logic behind that usually goes like this: if your requirements change, you can reflect this change in the next sprint backlog and this will lead to reduced cost, because designing the new feature and implementing it is close together in terms of time, so the cost goes down, according to the famous rule that the later you need to make a change to your requirements the more expensive it will be to satisfy that requirement.

But mid to big software projects are complex. A sudden change to requirements does not mean you won't have to touch other parts of your system in order to satisfy that requirement. In a lot of cases the architecture will need to be modified very significantly, which will also mean you'll need to re-implement all the features that relied on the older architecture. So the whole point of reduced costs kinda goes away here. Of course if a new requirement calls for a new independent part of the system, that's not a problem, the old architecture just grows, it doesn't need to be rethought and reimplemented.

And the opposite. If you are using waterfall and you suddenly realize that a new requirement has to be introduced, you can go and change your design. If it requires that existing architecture is altered, you redesign it. If it does not really mess with it but just introduces a new part of the system, then you go and do all the work, no problem here.

With that said, it seems to me like the only advantage agile development has is working feature complete builds between sprints, and for a lot of people and prjects this is not critical. In addition, agile seems like it results in bad software architecture overall, because features kinda get slapped one onto another, agile teams only care that a feature works, not how it works. This seems like that when systems grow in complexity with time, agile development practices actually increase chaos in the overall product architecture, thus eventually resulting in higher costs, since it will be increasingly more difficult to introduce changes, whereas waterfall allows you to perfect your architecture before you release anything.

Can somebody please point me as to where I'm going wrong here, because obviously a lot of people use agile in production environments, so I must be wrong somewhere.

Answer 1

First off, the "waterfall" model was always a straw man describing how NOT to run a software development project. Look it up.

Anyhow, most "waterfall" SDLC projects entail a "Change Management" process, for when people discover that the assumptions that were written into the specifications are no longer valid (and this always happens on large complex projects). Unfortunately, the change management process is built as an exception handling measure, and is stupidly expensive, meaning the project will end up late, or of poor quality.

The idea behind Agile methodologies is that change is a given--it will happen. Therefore, you must make "change management" standard operation rather than exception handling. This isn't easy, but folks have found that if you use a lot of good design practices, you can do it.

Another major problem with front loaded design is that (most often) so much time is spent in requirements gathering and design, development and testing time suffers. Also, if testing is the last phase, and a serious problem is discovered, it is highly unlikely to get fixed within your time frame.

Perhaps one of the best features of an Agile approach is that it demands continued interaction (that is, real communication) between the team developing the solution, and the customer who needs it. Priorities are made, so that the highest priority items are done first (and if time runs out, it's the low priority items that are cut). Feedback comes quickly.

Going back to the question of dramatic changes--you really need to use methods that mitigate changes. Good SOLID OO principals can play a good part of this, as can building solid automated test suites so that you can regression test your software. You should do these things regardless of your methodology, but they become truly essential if you are trying to be agile.

Answer 2

Well, there's a few advantages. The first is that customers don't read spec documents. Ever. Waterfall assumes that everyone will nicely agree to a spec at the outset and then when the software is delivered to the customer a year later matching spec, they'll be happy. In practice, customers only ever discover all the things they really need, really don't need, and really need to be something different when they are actively playing with the software itself. Agile gets a working prototype in customers hands, so these disconnects are caught immediately. Agile isn't just about reacting to changing requirements. It's also about having those changing requirements occur early, when the cost of change is low, not at the end, when the cost of change is high.

A second advantage is that because Agile has highly visible deliverables often, projects are less likely to go off the rails. With a large Waterfall project, it is all too easy to get way behind without even realizing it. Waterfall produces multimonth death marches at the end of the project. With Agile, because you are delivering to customers every couple weeks, everyone knows exactly where the project is and slips are caught (and adjusted for) quickly. In my experience, Waterfall produces weeks or months of 100 hour weeks at the end. Agile means that you might have to put in a couple 12 hour days at the end of the sprint.

A third advantage is that Agile projects tend to be more motivating. It is very hard for people to have any sort of sense of drive in a year long project. The deadline seems so far away and that lack of immediacy means that people tend to procrastinate, over-polish designs and otherwise just not work very productively. This is especially true because the initial months are spent on things people don't like to do, like spec documents. Because Agile always has a very immediate, concrete deadlines in the very near future, people tend to be more motivated. It is much harder to procrastinate with a concrete deadline for a fixed set of tasks due next week.

Answer 3

In response to the quote from the question, which is a fundamental misconception of anti-agile opponents

"... whereas waterfall allows you to perfect your architecture before you release anything." is a fallacy, let me fix it for you; "... whereas waterfall allows you to perfect your architecture so you never release anything."

The implication that Waterfall some how provides higher quality architecture is fundamentally false and completely disproved by empirical historical evidence.

If Facebook was designed in a waterfall fashion with 500 million users as a hard and fast requirement and not released until an architecture to support that was perfected no one would have ever heard of it today.

Same with YouTube, Twitter and countless other companies.

They got something that the customer could touch and respond to working and released it as quickly as possible. They got feedback and refined it and released it again; iterate. This is how in these three examples, they respond with only features customers accept and want and waste as little time as possible on things that customers find of little or no value.

Anyone with any significant years of software development experience will agree there is no such thing as a perfected architecture. Just one that starts out farther from entropy and big ball of mud than other alternatives. Agile acknowledges this and takes it into account. Building into the process, this as technical debt, rapid re-prioritization and refactoring.

Answer 4

Scrum by itself does not prescribe any technical practices because it is a general framework.

Agile software development requires technical excellence from the team. This comes from following technical practices from XP (extreme programming). For instance, you must learn about refactoring, tdd, whole team, pair programming, incremental design, ci, collaboration etc.

Doing it like that makes it possible to handle change quickly and safely, which is also the primary reason for using agile development in the first place.

The one measurement of agile that matters is if you regularly (weekly, biweekly) manage to release valuable, working software to your customer. Can you do that? Then you're agile in my book.

Answer 5

For me, the main benefit of agile is reducing risk in the project.

By delivering iteratively and getting lots of feedback from your user base, you're increasing the chance that you'll deliver something they actually want, and you'll do it by pragmatically delivering the most important features for them as early as possible. In theory, this will be done with better estimates and planning. This is obviously very appealing from the business perspective - much more so than just the releasable build.

You could argue that this constant change compromises your system and reduces quality, but I would say two things. 1) This change happens anyway on more traditional software delivery projects - it's just unaccounted for and occurs later on in the project which is when, as you pointed out, things are harder and more expensive to change. 2) Agile has a lot to say about dealing with this change in terms of using experienced motivated people and practices such as TDD, pairing and code reviews. Without that shift in mindset towards quality and continuous improvement, I accept that frequent change that agile implies could start to cause problems.

Answer 6

In a lot of cases the architecture will need to be modified very significantly, which will also mean you'll need to re-implement all the features that relied on the older architecture.

If your architecture needs to be modified significantly as a result of a requirements change, you have either bad architecture or a bad requirements. Good architecture allows you to defer as many decisions as possible and decouple components of your system. Good (user) requirements aren't things like "use a different database".

So, let's operate under the assumption that we have a good working architecture. If that is the case, then agile development methodologies lose this "wash" with big-up-front-design methodologies. After all, a core feature of agile development methodologies is practices like TDD that promote loose coupling in code, allowing the project to continue at a high velocity whether it's near the beginning or very mature.

Answer 7

In a lot of cases the architecture will need to be modified very significantly, which will also mean you'll need to re-implement all the features that relied on the older architecture. So the whole point of reduced costs kinda goes away here.

Following an agile process, there is a better chance of catching this requirment before too much software has been developed (And needs to be changed.). When you go several months without working with the client and giving them something to look at, you catch these major architectural problems only after you "think" you're ready to deliver.

I'd rather try to implement these changes in a working application that has test coverage than a massive pile of code that doesn't even compile. While being head of technical support, I was handed a CD of an application that had been in months of development and it wouldn't even install. We could have found that problem the first week, but instead it was time to order diner in because it was a long night.