Challenges to the Agile approach on government projects

Question

A previous Agile discussion here had good answers specifying what is critical to the success of implementing Agile methodology in software development. Most of the points were the typical organizational and management challenges, but one point worries me and it is that the client must be involved throughout the process.

The client is the one thing that you cannot control realistically, perhaps your business model gears you to government contracted work for instance where an intensely strict contract obligates the company to:

• Provide X features exactly as requested

• Feature requests will be thrown over a wall, don't bother us we don't want to hear it.

• There is no concept of feature priority in the customer's mind, they are all important or we wouldn't have asked for them.

• The project will cost no more and no less than Y regardless of overruns or deadlines.

• Absolute, strict, final and non-negotiable deadline for complete delivery of all work.

We have never worked with such a client before but the money on the project is just too good to pass up. We need this work.

I came here and worked HARD to change processes within to move towards Agile development and here I don't know how to reconcile where this project fits into our new process. I have never before had the luxury of open-minded hands-off management that trusted me to lead the development team and processes down this path and now that we are here I can't honestly tell myself that this project will truly be done in an Agile way. I feel like management trusted me to lead this path and that I let them down because this situation we are in now so clearly calls for Waterfall. I am afraid that I might lose their trust if I backtrack now.

Other answers like the one here say Agile is impossible with this kind of client, do you agree? Have any of you been in a similar situation and made it work? What strategies did you implement to make Agile happen successfully?

Answer 1

I think the first thing to realize is that there is a difference between being Agile and being agile. Slowly rolling out agile techniques and characteristics - cross-functional teams, adaptive planning, evolutionary/incremental delivery, time-boxed iterations, and even introducing concepts from Lean are very different than introducing Extreme Programming, Scrum, or Crystal.

You explicitly mention customer involvement. Yes, many of the Agile methodologies call for customer involvement, but that's not required to be agile. In every government/defense related program, I've always had a program or project manager who was the point of contact with the customer. This person becomes the "voice of the customer". It might be slowed down as they teleconference or email or call and clarify, but you can have a single person (or a group, if you have deputy PMs as well) that is the customer representative of your team. Admittedly, it's not quite the same. But isn't being agile about being flexible and responding to change?

You also mention a few key concepts: predefined requirements, having feature requests "thrown over the wall", a lack of prioritization because "they are all important", and fixed-cost and/or fixed-schedule projects. Each of these can be addressed in different ways.

If you think you have all of your requirements up front, chances are you don't. Requirements do change. Just because you have a "finished and signed off" specification doesn't mean it is set in stone. Given whatever requirements document you have, capture them how you feel comfortable and/or in the manner specified by the contract and deliver the requirements, the design, and the architecture. In addition, see if you can treat these are living documents (a design document I saw today at work is labeled as Revision G, which means it's on it's 8th update). Ask about how much you can leave as TBD in any given iteration and how much needs to be firmed up now - there might be some give and take.

Be agile with your documentation. Don't duplicate efforts between "what your team wants" and "what the customer wants". For example, if your customer wants a traditional software requirements specification and your team wants to use user stories, try to adapt to a traditional SRS and use action items and a rolling action item list instead of user stories so that you don't spend time formulating both "the system shall..." and " must be able to because ". This does take discipline on the part of the team, though, to adapt to differences between projects. Capture problems in reflections.

Once you get to development, you might run 5 or 6 iterations, and then invite your customer to your facility to see a subset of your implementation. Rinse and repeat this process. It's not the constant involvement demanded by some methodologies, but you do have the advantage of high visibility. If your customer says no, at least you tried. If they say yes, you can enlighten them on being agile. On one project I was on, the customer visited the site every few months (3-5 months, usually). They would watch us go through QA testing, they would discuss concerns with engineers, and business with the program/project office. It was an opportunity for everyone to get on the same page.

Testing and maintenance happen the same as on other agile project. Create your test procedures and document defects in the appropriate way, track metrics per contractual obligations, and document test results. If you want to follow TDD, go for it. Continuous integration is another good idea. During project status meetings, your project manager can use this information to say "we implemented N requirements, have M tests, X tests pass" and update on project health and status to the people with the money.

Speaking of money, we have the fixed-cost and/or fixed-schedule problem.

Dealing with a fixed schedule is fairly straightforward. Given your requirements, you know how many iterations that you can complete. Your workload for each iteration is pretty much set in stone in terms of features to implement, test, and integrate. It might be difficult, but it's not impossible to break up features and assign them to iterations in advance. This goes back to my point about inviting the customer - if you have one year and are using 2 week iterations, perhaps invite the customer quarterly (and invite them every quarter) and show them the results of the previous work. Let them see your prioritization of requirements, your future plans, and how you are going about scheduling.

Dealing with a fixed budget is similar. You know how much time you have, how many resources you have for the project, how much they cost, and therefore how many hours everyone can work per iteration. It's just a matter of ensuring that everyone keeps track of this carefully. If your company can eat the cost of overtime, go for it. Otherwise, make sure everyone works the appropriate length of time and use good time management skills and time-boxing to keep everyone productive. More productive hours is what you need to keep costs down - deliver more value-adding documents and software without the cost of meetings and overhead.

Ultimately, it's not about necessarily being Agile, but applying the things that make Agile good and being agile. Be able to respond to changes in requirements, be able to deliver frequent software even if the customer doesn't want it, only produce value-adding documentation (along with whatever you are contractually obligated to produce), and so on.

Answer 2

Yes, agile isn't appropriate for such a project, because it sounds like the requirements are already done and fixed in stone, probably the result of years of analysis by expensive consultants, committee meetings and political compromise. Waterfall works fine if the customer's so disciplined that they can tell you in writing exactly what they want. They might be wrong, but at least you have it in writing, and you'll get paid if you deliver that. (This says nothing of customer satisfaction, of course. Chances are you'll deliver something that they don't actually need.)

Agile was created to solve a problem that you don't have: risk due to uncertainty in the requirements.

It's true that the customer might ask for change requests, but you either follow one of two paths:

1. The money was so good that you know you can absorb X amount of new features at various stages of the project and still come out without losing your shirt, or
2. You make it clear to the customer at the outset that due to them negotiating a tight price, that each new feature request will generate a change order with a price that will have to be approved by them, with an accompanying purchase order (or amendment to the original purchase order) in order for you to implement that. The change order will spell out any impacts to the functionality or schedule. If they say the deadline can't be moved, then change orders just become exponentially more expensive as the project progresses because it's more expensive to change stuff later on.

The relationship feels a lot friendlier under situation #1, but the fact is it's pretty rare to find purchasing departments that won't squeeze you on the price, so most of the time you're in situation #2. That means the relationship is confrontational, but if you want to survive in business, you have to get good at managing the relationship while holding your ground. This is a large part of the project manager's job.

It sounds like you might be in situation #1, which is good. I imagine that government contracts are the only place where they don't care about money, because, after all, they're not spending their money, they're spending your money.

Answer 3

...government contracted work for instance where an intensely strict contract obligates the company to:

First. It's strict. But it's not inflexible. It simply requires attention to detail and a long, long string of change orders.

Government agencies actually are agile in a slow, inefficient way. You have to write (and negotiate) formal, detailed change orders all the time.

Provide X features exactly as requested

Until modified by a change order.

Feature requests will be thrown over a wall, don't bother us we don't want to hear it.

The channel of communication is the change order. Budget and Schedule Impact.

There is no concept of feature priority in the customer's mind, they are all important or we wouldn't have asked for them.

This is hard to work around. Even non-government businesses that spend a lot of money for "requirements analysis" don't want to be told that a big, flat, steaming pile of requirements unencumbered by priority and tradeoff information is incomplete. They paid good money to get all the requirements. How can you want more information?

This is a difficult problem.

The project will cost no more and no less than Y regardless of overruns or deadlines.

Except for the change orders. Which modify Y and the Deadline.

Absolute, strict, final and non-negotiable deadline for complete delivery of all work.

"non-negotiable" is generally not true. It's negotiable. It's just painful to negotiate.

The important part of negotiating with government agencies is the fact that you need "lawyer-level evidence" for your cost and schedule changes. A few careful technical presentations with a nice power-point slide isn't "evidence". You need a lot of documentation to make your case.

The government folks need to provide unimpeachable evidence that they've done everything in their power to make this as cheap and effective as possible. They know that every decision is replayed in the public press and scrutinized in hind-sight.

The complexity of software development, and the after-the-fact, "monday morning quarterback" aspect of government work makes them reluctant to make changes to the contract without overwhelming evidence.

It makes a properly Agile approach difficult.

"Individuals and interactions over processes and tools" is tough. You aren't working with an individual, but a representative of the government who's job is constrained by process.

Answer 4

In a project like this, they have tied your hands on scope, resources, and time. The only thing you have left to manage is quality. So...

You aren't going to get most of the benefit from an agile approach that you might otherwise, but you can do your best to mitigate quality risks and be able to inform the client of problems earlier rather than later.

So be as agile as you can:

1. Go through the requirements and prioritize them best on technical risk. Set the prioritized requirements as your backlog.
2. Work through the backlog in sprints--design, test, and code the features for the sprint. You're missing client interaction, so the requirements document has to represent the client for this activity.
3. Invite the client to each sprint review--they can say no, but they might say yes. And you'll get feedback sooner rather than later.

If you start running against the deadline, you'll be able to show what's done, and perhaps at that point the client, knowing he's not going to get everything, will prioritize enough to tell you what he does want. You should have the riskiest stuff done, too, meaning that the tasks at deadline time are the easiest to cram in working extra hours.

Answer 5

I think this type of client is not the norm. You're dealing with a group that has requested similar projects before, so they do know exactly what they want. You never mention that their specifications will change.

Provide X features exactly as requested

I'm lucky if I provide X feature vaguely as suggested and be ready to change it at a moments notice.

Feature requests will be thrown over a wall, don't bother us we don't want to hear it.

If you know what they want, go and build it.

There is no concept of feature priority in the customer's mind, they are all important or we wouldn't have asked for them.

You can't lose on this one. Build them as you see fit.

The project will cost no more and no less than Y regardless of overruns or deadlines. Absolute, strict, final and non-negotiable deadline for complete delivery of all work.

That's a tough one if you've never built a project for the gov. If you have some history, you may be able to determine if you can deliver. This doesn't mean they don't pay well (they're notorious for paying $50 for a $10 hammer.) or have unreasonable expectations. With these specs, someone on your team should act as the customer and approve the work compared to the specs. Even if you found a flaw and begged them to change the requirements, they probably wouldn't.

Answer 6

Sadly what you've described is the typical customer view of how a software project should be tackled. This is not to say that the customer is being unreasonable; after all -- isn't this how one would execute the building of anything else (a house, for instance?). That being said though, I'm not really offering anything more than what we all already know. What you're asking is ... is application of agile practices feasible in this situation?

I have the benefit of having just finished a project that is similar in many respects to the situation that you describe, i.e.,

1. Fixed (in stone, come hell-or-high-water) deadline.
2. Functional Requirements Document ("the bible"). Unsurprisingly inadequate.
3. Traditional roles: Project Manager, Business Analyst.
4. Weakly engaged business users (can you say "no product sponsor"?).

... and of course, the forward-thinking development team is trying to work in an agile fashion, despite the above:

1. 2 week iterations;
2. Stand-ups;
3. Retrospectives;
4. Pair programming;
5. TDD;
6. Continuous integration.

Is any of this remotely meaningful to Business though? No. Two months to deadline, the up-until-then carefully observed iterations and planning meetings are abandoned in a frenzy of headless-chicken-itis.

The answers that others have provided above are to a greater or lesser degree compromises. In my opinion, agile (whether "Agile" or "agile") is "done in" in a pernicious fashion when we compromise. In my opinion:

There is no compromise, or there is no agile.

The very spirit of agile is about cutting to the chase, removing the waste, being brutally honest with oneself. It is a now well-documented and undeniable fact that software estimation on large projects is a gamble at best. Is it not our duty as software professionals to educate prospective clients of this? If clients are unwilling to accept that we're the experts, then isn't it our professional duty to walk away?

Answer 7

When I started working where I am now, I found myself asking the same question you asked quite a lot. There is something to be said for waterfall with government contracting. Ironically, agile has now become a buzzword with government customers (who realistically work in a waterfall manner), so now we are left trying even harder to implement an agile process with a basically inflexible customer.

We have a system that has been described as "Scrummerfall" "Agilefall" or "A mess", but in many respects we have slowly been able to adopt a more and more agile process as this (gargantuan) project has moved forward over the years. One of the ways that we did that is basically finding back channels of communication with the USERS of our system, as opposed to our CUSTOMERS. Our customers are a stuffy department headed by appointed officials who will never touch our software in their working lives and don't want to understand it. Our users are regular government personnel in the field attempting to accomplish an important task. For us, the key to establishing a communication feedback loop that enabled us to be as agile as we are was the necessity for UAT (User Acceptance Testing).

At an early point in our project, it was decided that a representative group of ACTUAL USERS from various offices of our vast government customer would be gathered on location HERE and we would have a couple weeks of facetime with them as they run through a series of test scripts to test out our software. Very much as an informal thing, the requirements team turned this time into an invaluable way to get feedback from actual end users. Meanwhile, the UAT test team within the government worked through their bureaucracy to have more and more influence over the formal requirements process on their end including change orders. The end result is that BAs like myself act as stand-in product owners embedded in scrum teams and are able to get invaluable face time with real customers that allow us to function in a very agile way.

Obviously, there are a lot of issues, and we are still not really agile, but we are agile enough to have been held up as an example of a major agile project actually using that methodology in the government contracting sector.

To sum up: use your experience as an agile evangelist within your own organization to infiltrate your customer. Go through a learning process with them, establish a partnership based on trust with key people on their side, and work AROUND the formal, ossified Requirements Process that they inevitably have in place. You will be thanked by the guys on the ground who have to actually use what you are developing!

Answer 8

You are assuming the requirements are well written and you think they mean what they think they mean. The back and forth of the agile process will help ensure they are getting what they meant in addition to what they asked for.