Can agile be effectively used for initial development on a substantial project?

Question

I think the short answer to this question is probably "yes", but I thought I'd ask anyway!

We are in the process of taking a number of existing similar-but-different applications and, as part of what is for us possibly the largest development project ever, attempting to replace them and any subsequent similar-but-different applications with one over-arching super configurable system to rule them all. I fully expect Difficult Second System Syndrome to give us plenty of problems delivering the behemoth in the first place, but I don't want our choice of methodology to make it all worse. We've dabbled in agile, from something mostly waterfall, but this would be our first substantial project using some new improved methodology.

We have Big Plans for the new system, and while it doesn't all have to be delivered at once we will need to implement large chunks of it from the word go to allow anything to be given to the customer that they would even understand. My concern is that, since we're dramatically changing so much at once, there is little scope for small chunks of work that lead to useful/demoable deliverables. I'm worried that, even if we did manage to find a way to do it in small chunks, we'd need those chunks to fit well into a bigger picture, and so would find it hard to develop them in isolation for that reason too. We're aiming for quite a decoupled SOA approach, which is something.

Timescales are tight, and reputations have been staked on delivering on time.

So I'd like to know if, in this case:

• it is advisable to do loads of up-front design, put together the overall shape of the app and get the crucial modules working in one big push (which might be sprintable in terms of keeping track of work, but not so much in terms of feedback and improving design and so on), before maybe going more agile from then on, or...
• try to be as agile as possible from the start - agree things like coding standards, maybe some API basics, general module splits, and then trust to independent coding while bearing flexibility in mind to let the other stuff gradually flesh out; maybe some stuff has to be reworked, but anything that can be better steered during development rather than up-front will benefit, or...
• maybe something else, and...
• any general tips?

Answer 1

Yes... If...

I'll only say yes if you can drop features from your project to hit the deadline.

Otherwise you will only be getting a portion of the benefits from an Agile Approach.

Arbritrary Deadlines don't work

Timescales are tight, and reputations have been staked on delivering on time.

This worries me, this smacks of an arbritrary deadline created from optimistic best guess analysis. Arbritrary deadlines do not work, and in fact have been proven to work in detriment to a project.

Joel Spolsky wrote a great essay on Evidence Based Scheduling

3) Simulate the future

Rather than just adding up estimates to get a single ship date, which sounds right but >gives you a profoundly wrong result, you’re going to use the Monte Carlo method to >simulate many possible futures. In a Monte Carlo simulation, you can create 100 possible >scenarios for the future. Each of these possible futures has 1% probability, so you can >make a chart of the probability that you will ship by any given date.

Be prepared to drop features to save time

Putting that aside having a fixed timeframe for an Agile Project is fine, however you need to be able to prioritize your most valuable features in your software product and then drop your not so valuable features to reach your time milestone.

This should happen every single sprint, the only features that get merged into a "release" at the end of a sprint are features that are fully tested and functioning. Features that are not fully tested and functioning get dropped for the next sprint or next "release". Release doesn't mean deliverable to the customer, but after every sprint you have a fully working (albeit in a limited fashion) software product that is fully tested.

Use the best features of Agile

The idea of using agile is that your features between sprints is not set in stone. In a normal agile project the customer has input at every sprint and may request a new feature be added as a higher priority than a feature requested 3 sprints ago.

Avoid a project race (if at all possible)

In your case this probably isn't going to happen, you already have the feature set out in your existing projects, and in addition it also sounds like you are going to have a race condition whereby you are going to be racing your current maintenance cycle for your existing projects with your new project so that you can replace it.

• http://butunclebob.com/ArticleS.UncleBob.CleanCodeArgs

If you have been a programmer for more than two or three years, you have probably been significantly slowed down by someone else’s messy code. If you have been a programmer for longer than two or three years, you have probably been slowed down by your own messy code. The degree of the slow-down can be significant. Over the span of a year or two, teams that were moving very fast at the beginning of a project can find themselves moving at a snail’s pace. Every change they make to the code breaks two or three other parts of the code. No change is trivial. Every addition or modification to the system requires that the tangles, twists, and knots be “understood” so that more tangles, twists, and knots can be added. Over time the mess becomes so big and so deep and so tall, they can not clean it up. There is no way at all.

As the mess builds, the productivity of the team continues to decrease, asymptotically approaching zero. As productivity decreases, management does the only thing they can; they add more staff to the project in hopes of increasing productivity. But that new staff is not versed in the design of the system. They don’t know the difference between a change that matches the design intent, and a change that thwarts the design intent. Furthermore, they, and everyone else on the team are under horrific pressure to increase productivity. So they all make more and more messes, driving the productivity ever further towards zero.

Eventually the team rebels; and they inform management that they cannot continue to develop in this horrific code base. They demand a redesign. Management does not want to expend the resources on a whole new redesign of the project, but they cannot deny that productivity is terrible. Eventually they bend to the demands of the developers, and authorize the grand redesign in the sky.

A new tiger team is selected. Everyone wants to be on this team because it’s a green-field project. They get to start over and create something truly beautiful. But only the best and brightest are chosen for the tiger team. Everyone else must continue to maintain the current system.

Now the two teams are in a race. The tiger team must build a new system that does everything that the old system does. Not only that, they have to keep up with the changes that are continuously being made to the old system. Management will not replace the old system until the new system can do everything that the old system does, on the day of the change.

This race can go on for a very long time. I’ve seen it take 10 years. And by the time it’s done, the original members of the tiger team are long gone, and the current members are demanding that the new system be redesigned because it’s such a mess.

If you have experienced even one small part of the story I just told, then you already know that spending time keeping your code clean is not just cost effective; it’s a matter of professional survival.

An Alternative - clean up your existing architecture

Robert Martin as quoted above expresses an alternative to a project rewrite in his talk about software craftsmanship. If you haven't seen this before its well worth watching all the way through.

• http://www.infoq.com/presentations/craftmanship-ethics

I also recommended some methods to perform gradual refactoring as opposed to a complete rewrite in my answer to a similar question.

• What is the best/ethical way to estimate code clean-up?

Getting Started

In the past where I have started large greenfield projects (unsurprisingly it was also SOA). I have found that the initial architecture prototyping and bedding down of the framework took a bit of time where I may redesign my approach, scrap my ideas half a dozen times over as I prototype and test out theories and best design practices.

Don't waste time writing specifications that will only be thrown away

Using a big up front design in a new architecture will waste time and effort so taking an agile approach of functional requirements and specifications over a fully fleshed out specifications document is essential to not waste time and throw away bad ideas that looked great on paper but didn't work out in real life.

Initial Prototyping

This initial prototyping stage I have found generally doesn't produce anything that is deliverable to the customer until its actually finished and working. However what you should end up with at the end should be a well thought out architecture, with easily configurable modules and refactoring the existing applications into the new architecture should lend itself quite easily to an agile methodology with customer visible results.

Followed by a ramp up in velocity

Once your prototype is finished and your architecture bedded down you will find that you will have a transition period where you will need to educate the rest of your team as to how best to migrate or refactor your existing projects to fit in with the new framework. So you can expect a slow ramp up speed followed by increased velocity nearing the end of the project as you get more developers online and implementing your design.

Answer 2

I think the classic Rational Unified Process works better for big projects.

The reason I say this is that the requirements gathering phase for a large project takes a long time, and, the requirements interact in unforeseen ways. For instance someone in accounts will tell you they need the Tax Id. of anyone paying a bill because of money laundering regulations -- you then have to go back and redesign your Web site to capture Tax Ids. in the registration page, and also, end up with two extra forms if the name on the Credit Card doesn't match up with the registered customer name.

In many cases it is just not possible to drop features. You cannot just go live with "Order Capture" after dropping the "Order Delivery" feature.

Once you get away from single purpose applications and start developing "real" business applications you find that most of the requirements are actually regulatory. You need to do stuff because of Accounting, Tax, Consumer Protection, Privacy laws etc.. In finacial systems this can account for up to 100% of your requirements for some systems.

The agile "lets talk to the users and feed them cookies" is just not possible when your "user" is represented by a document with two hundred pages of obscure legalese. If you drop features like "Report Sales Tax" you may end up in jail.

Perversely, given the above comments, my answer is "Yes" you can use Agile and "Yes" it works -- but only for "Sub"-projects within the larger project framework, and, only if you take "go live" to mean delivered to "UAT" or "SIT" test environment.