How do you explain to an "agile" team that they still need to plan the software they write?

Question

This week at work I got agiled yet again. Having gone through the standard agile, TDD, shared ownership, ad hoc development methodology of never planning anything beyond a few user stories on a piece of card, verbally chewing the cud over the technicallities of a 3rd party integration ad nauseam without ever doing any real thinking or due dilligence and architecturally coupling all production code to the first test that comes into anyone's head for the past few months we reach the end of a release cycle and lo and behold the main externally visible feature that we have been developing is too slow to use, buggy, becoming labyrinthinly complex and completely inflexible.

During this process "spikes" were done but never documented and not a single architectural design was ever produced (there was no FS, so what the hell eh, if you don't know what you are developing, how can you plan or research it?) - the project passed from pair to pair, each of whom only ever focused on a single user story at a time and well the result was inevitable.

To resolve this I went off the radar, went (the dreaded) waterfall, planned, coded and basically didn't swap off the pair and tried as much as I could to work alone - focusing on solid architecture and specifications rather than unit tests which will come later once everything is pinned down. The code is now much better and is actually totally usable, flexible and fast. Certain people seem to have really resented me doing this and have gone out of their way to sabotage my efforts (possibly unconsciously) because it goes against the holy process of agile.

So how do you, as a developer, explain to the team that it is not "un-agile" to plan their work, and how do you fit planning into the agile process? (I'm not talking about the IPM; I'm talking about sitting down with a problem and sketching out an end-to-end design that says how a problem should be solved in sufficient detail that anyone who works on the problem knows what architecture and patterns they should be using and where the new code should integrate into existing code)

Answer 1

I think (and I may be going out on a limb here) that ALL projects should have a bit of classic waterfall: The initial analysis and specification phase is essential. You must know what you are doing, and you must have it in writing. Getting the requirements in writing is difficult and time consuming, and easy to do badly. That's why so many skip it - any excuse will do: "Oh we do agile so we don't need to do that." Once upon a time, before agile, it was "oh I'm really clever and know how to solve this, so we don't need to do that." The words have changed a bit but the song is essentially the same.

This is of course all bull: You have to know what you are to do - and a specification is the means by which developer and client can communicate what is intended.

Once you know what you have to do - sketch out an architecture. This is the "get the big picture right" part. There is no magic solution here, no one right way, and no methodology that will help you. Architectures are the SYNTHESIS of a solution, and they come from partly inspired genius, and partly hard-won knowledge.

At each of these steps there will be iteration: you find things wrong or missing, and go fix 'em. That's debugging. It's just done before any code got written.

Some see these steps as boring, or not needed. In fact, these two steps are the most important of all in solving any problem - get these wrong and everything that follows will be wrong. These steps are like the foundations of a building: Get them wrong and you have a Leaning Tower of Pisa.

Once you have the WHAT (that's your spec) and the HOW (that's the architecture - which is a high-level design) then you have tasks. Usually lots of them.

Bust the tasks up however you want, allocate them however you want. Use whatever methodology-of-the-week that you like, or that works for you. And get those tasks done, knowing where you are heading and what you need to accomplish.

Along the way there will be false trails, mistakes, problems found with the spec and the architecture. This prompts things like: "Well all that planning was a waste of time then." Which is also bull. It just meant you have LESS foul-ups to deal with later. As you find problems with the high-level early days stuff, FIX THEM.

(And on a side issue here: There is a big temptation I've seen over and over to try to meet a spec which is expensive, difficult, or even impossible. The correct response is to ask: "Is my implementation broken, or is the spec broken?" Because if an issue can be sorted out quickly and cheaply by changing the spec, then that is what you should do. Sometimes this works with a client, sometimes it does not. But you won't know if you don't ask.)

Finally - you must test. You can use TDD or anything else you like but this is no guarantee that at the end, you did what you said you would do. It helps, but it does not guarantee. So you need to do final test. Thats why things like Verification and Validation are still big items in most approaches to project management - be that development of software or making bulldozers.

Summary: You need all the up-front boring stuff. Use things like Agile as a means of delivery, but you can't eliminate old-fashioned thinking, specifying, and architectural design.

[Would you seriously expect to build a 25-story building by putting 1000 laborers on site and telling them to form teams to do a few jobs? Without plans. Without structural calculations. Without a design or vision of how the building should look. And with only knowing that it is a hotel. No - didn't think so.]

Answer 2

I'm still amazed that many people think that TDD means writing unit tests first. No it means writing tests you will need before you write the code. The test actually can be unit test, integration test, end-to-end test and of course performance test (well you will probably not write performance test before the tested code but it doesn't mean you should not write it at all). The needed type of the test should be visible from the definition of done for the user story. If one of acceptance criteria for the user story says that feature should provide the result within 500ms with 50 concurrent users then user story cannot be considered as completed until you have performance test which will prove that this acceptance criteria is satisfied. Once you have the automatic test you can check every day that it is passing as you add other features.

It more sounds like your acceptance criteria wasn't defined correctly and because of that you couldn't test your performance. Still it can happen that application perform badly once you deploy it to the real environment and use it under heavy load but it can be always considered either as failure of requirements (if requirement is not defined developer doesn't take it in mind when working on the code) or development team (insufficient testing against defined requirements).

Another interesting thing is that developers in your team are focusing on single user story. Agile is about communication so developers should communicate what user stories require and how it affect the rest of the application - collaboration is a must. Test should cover that as well so if one developer breaks the needed functionality for another user story it should be visible in automated tests. If you still feel that it is not enough or it doesn't work you can introduce architecture meeting where the whole team cooperates and discuses architecture of the application. It is usually enough to have such meeting once a week.

A lot of things from waterfall process are replaced with communication and automatic tests. Nobody says that you cannot write any high level documentation or design! You can and many teams use for example Wiki for that.

Answer 3

[our product was] too slow to use, buggy, becoming labyrinthinly complex and completely inflexible.

That has nothing to do with agile, it's a sign of programmers not doing their job properly.

One basic idea of agile is that the team has complete control over the process. That means they must agree on the amount of planning, documentation, testing and how they deal with refactoring. All that power is great, but it also comes with responsibility and this is probably where your team failed. They accepted their freedom, but neglected their responsibilities.

In the end it's just a matter of explaining about code quality and trying to agree on a way to increase it and keep it that way. Normal programming practices apply, really.

Pro tip: use your "Definition of Done" to enforce this. Make sure everyone agrees on it and place it visible for everyone. Use it as a strict gatekeeper for deciding if a story is completed or not. It's even possible to add non-functional requirements (like performance) to that list as well.

Answer 4

Yeah. Your teammates are idiots. Your project sucked because of Agile. Feel better? Okay, let's move on. :P

I think you and your team will be able to communicate more effectively about what went wrong if you concentrate less on the names of your capital-M Methodologies and more on why the software you wrote was so slow and buggy. Don't use the terms Agile or waterfall at all. They are clearly emotionally loaded in your office.

Agile works sometimes. It didn't work for your team this time. Some people will say that's because you did Agile wrong. After all, Agile works, so if you had done it right it would have worked, right? Whatever.

It doesn't sound like anyone disagrees there was a failure, but you aren't going to win friends, influence people, or do any better next time if you blame a methodology. That probably had little to do with what actually went wrong.

Instead, concentrate on pinning down the most direct causes of the failure and work with the team to make sure they don't happen again. This will require people skills.

Speaking of people skills, you shouldn't be surprised that your team didn't appreciate you making them look bad by doing all their work and doing it better than they did. By doing this "under the radar" you may have damaged some relationships you'll now have to rebuild to be an effective member of the team.

I think the best way to look at a situation like this is to consider the long-term total output of the team as a whole. You may have saved the week this time, but you might have done better in the long run for your company by building a better team.

I'm telling you all these things, but I think you probably know most of them already and could explain them to someone else if you weren't so close to this situation.

Answer 5

If you want to add a pithy quote to your discussions, Eisenhower had a good one:

"Plans are nothing; planning is everything."

http://www.brainyquote.com/quotes/quotes/d/dwightdei149111.html

He meant that you should expect to change your plans and so don't put too much value in the plan as it exists, but at the same time the process of planning will sharply focus your energies in a crucial way. You have to make plans before you can test them and refine them.

Answer 6

+1 This is the best description of enterprise agile I have read recently.

Everyone who gets along with agile points out that it was never supposed to mean this, but once everyone gets caught up in the metrics this is exactly what you get from a team that has no passion about the quality of the product and either is obsessed with test coverage above all else or with meeting their weekly deliverable deadlines regardless of the corner they have painted everyone else into because that is all that makes it up to the management level on a weekly basis.

I have never seen this fixed with anything less than a re-org. If you are a mid-grade company with nothing special to attract really passionate people then even that won't fix it unless the next management changes methodologies sometimes.

Agile only seems to work well in orgs where the dev team cares enough to make a good product in spite of the extra, uncredited work that is required. Effectively they have to care enough to make it good on their own time, fight for improvement (and be willing to burn a lot of extra time redoing tests in some TDD cases)

You obviously care about shipping a good product, they obviously care about going through a scripted set of motions and receiving a paycheck to continuously clean up the mess they make.

I would look for less irritating employment elsewhere be it agile or not.

If you are completely burnt on agile there are lots of places that still do other methodologies. Almost anything on bid or contract for example.

Answer 7

I tend to use a sort of hybrid. The overall plan is waterfall, but there is agile in the execution.

My guess is that if the situation is as bad as you say, your team isn't really using agile, they're just being lazy or undisciplined. Agile isn't about not planning, it's just about being flexible, and, well, agile.

Answer 8

We have the same problems with some staff.

Basically "I don't know until I write it" is a favourite statement. So we went the opposite way around, we worked with the client to define the deliverables with sign off points. The last one being "now write the code to do X".

If we have "write design/testing/plan etc" in the same deliverable sprint as "do the fun and interesting code" then the first part never happens ...

BUT if I place "you don't get to do any fun and interesting code until you have said what your going to build and the client has signed off" then

• you firstly get grumbling acceptance, and a few tears and I had to do a lot of the filling in the blanks work and extra effort ...
• then you get dawning recognition when you ask them "so how was the process" that its better to have attempted the first version on paper
• then you have the test cases which the developers can see, and you can point exactly to the expectation of "what does done mean".
• then clients sign off on the development has an 80% less rejection rate ...
• plus you watch all the developers walking around holding the design documents and talking to each other over the designs ... they start to really get it.
• Then you work out how to break down the project plan into bite sized chunks and make a process out of it.

The agile part comes when the client then changes from the plan and you can adapt within a 2 week cycle NOT fly by the seat of your pants and make it up along the way.

In our case the "big design upfront" was broken into 9 stages (actual production releases) with an average of 5 substages. The designers and developers keep pace with each other, the designers being 1-3 substages ahead of the developers ... too far and discoveries in development break too much of the design, too close and tweaks to design cost to much as they have been already implemented "set in stone" within development. Each substage is 2-4 sprints worth for 1 developer.

In smaller projects we just have the same developer do design first > sign off > invoice > develop > sign off > invoice ... in cycles.

The test naming problem

The testing has many faces, formally there is about 7 catagories of tests each with sub sections ... one of these later ones is "write automated unit tests".

Its a bad name to have "test first development" purely because of developers understanding about what "tests" means in this context they see tests as write the actual code for the test against the interface for the implemntation. When its not really that at all ... you can do this when it comes to the writing the code but its really "validate the design against use cases and user stories BEFORE starting to write the code" ... done properly this removes many of the issues normally discovered during development while its still in the far cheaper "paper that can be thrown away" version.

Answer 9

One of the key elements of Agile is the idea of continuous improvement, and the idea that the entire team owns the project. So the correct approach would have been to review the issues with the team, and have the team decide how to correct it.

One team member going off by himself, ditching all of the principles of Agile and doing things his own way may result in a small amount of code working well but it doesn't really advance the whole project in a positive way.

Answer 10

One way to get them to work may be to make a T-Map that covers all of your sprint back-log

Each team has a thread, each sprint is a period. It all ties together (which is where your team seem to fall over). Pin this up somewhere, and get magnets to represent the pairs/subteams. They can even 'race'. But everyone knows where they and the other teams are. Put dependancies up on here if there are any.

The point here is you convey the total process but also break it down into sprints. Even if only the first sprint is put on there, and the other periods are blank this should be an excellent roadmap to finishing the project.

Answer 11

You have two problems: not enough shape on the requirements and poor architecture.

Requirements

You need both an overall end goal and the detailed roadmap of how to get there.

In the waterfall world, the end goal is the functional specification, and the roadmap of how to get there is the is the project plan.

In the agile world, one way is to capture it in an epic user story. Then the roadmap are the detailed user stories that flesh out the detail of the epic.

I've never been completely happy with that story, because the epic story never captures enough meat to get across the full idea. So what I've tended to use is a high level systems requirements document in conjunction with an epic user story or two. After that, the roadmap is the detailed user stories.

The nice thing about having a systems requirements document is that then the acceptance criteria for many of the user stories can be pulled out.

Think of the high level system requirements document as a "cut sheet" that marketing might use to sell the product to a technically savvy customer.

Architecture

One of the things that the "cut sheet" gives you is it puts boundaries on the system you're designing. That lets you make informed decisions, early on, about the architecture to use.

If your team had had such a document early on, you would not have to go through the pain of re-architecting the system later.

---

Actually, a third problem you have is poor communication. Communication is a two-way street (or multi-way between multiple people). However, that's just a human failing and takes (sometimes) superhuman efforts to get right.