How to get good design when using agile methods?

Question

I have been using an agile methodology (SCRUM) for about three years now and I see certain advantages to it, especially in the short-term feedback at many levels (from customers having early access to implemented features, from testers that can test features as soon as they are implemented, from other developers that can provide very early feedback on new code through review, etc).

On the other hand, I have two open problems, the first of which I will try to explain in this question.

Problem : Difficulty to get a good design

I try to do refactoring as soon as the code gets messy, I write unit tests as much as I can (which does help to prevent bugs in general and when refactoring in particular). On the other hand, developing some complex feature in small increments, with daily commits and continuously rethinking the code when it gets unstructured is not allowing me to produce really good design.

The only well-designed module I could produce recently I obtained by taking a different approach: I analyzed the problem for a few days (I actually had had the problem going through my mind for a couple of months before I started to work on it seriously), sketched a rather detailed design of all the involved classes and their relationships for a couple more days, and then locked myself up in my office and wrote down the whole code by working without interruption for about three weeks. The result was the best thing I have produced in a while, with very few bugs that were rather easy to locate and to fix, and with a very clear design, which has not required any relevant changes since then.

So up to now I found it much more effective to get an overall picture of what I want to do beforehand than starting to write code in small increments in the hope that the big picture magically emerges in the process. With my best effort, the small-increment development approach has always led me to worse design.

Question: Has anyone had a similar experience? Am I applying SCRUM the wrong way or what should I pay attention to if I want to develop in small increments and still end up with a well-designed piece of software? Or should I schedule a design user story before starting the actual coding? Is this considered a good practice, at least for features that are more complex than average?

EDIT - NOTE

I am aware of the fact that good design is not something absolute and does not have a value on its own but it depends on the context, and that one should aim at a design that is good enough for the problem at hand.

For example, I do not care (too much) about good design if I have to implement a simple component that (1) must be ready as soon as possible, (2) is going to be used only once, (3) is not used by other parts of the system (YAGNI).

I do care about good design when a component (1) will be used several times and in several different releases of a product, (2) needs to be maintained and extended over time, (3) has a lot of other components depending on it.

Answer 1

On the other hand, developing some complex feature in small increments, with daily commits and continuously rethinking the code when it gets unstructured is not allowing me to produce really good design.

Then don't do that.

Not everything fits into the nice agile box. Often times a product will have a few complex things that can't be farmed out to individuals and cannot be completed in any sane manner within a sprint. Forcing them into that box will only cause trouble. But these should be few and far between. If you find that many of your components are like this, your product owner needs to work at breaking things up better (with your team). I am fine doing like what you did: take the story, design it, then hammer it out ASAP with the expectation that it will take a few weeks.

In the more general case, there are 3 things that I've seen done for getting good design in Agile:

• Actually do design - A lot of places I've seen start their sprint and just start cowboying out code. You'll get bad design this way. Agile doesn't change the development process of plan - code - test - release, it just shortens it to more granular pieces. At the beginning of the sprint, sit down as needed and actually design your solution.

• Have an architect/lead - Once your project is big enough, you'll end up with multiple scrum teams working on different pieces of the application. It is very useful to have someone (or multiple people depending on the size of the application) whose primary job is to know what all of the teams are doing from a design standpoint. They can answer questions and guide the teams towards a more harmonious over-arching design. Having each scrum team have a lead that knows most of what other teams are doing I've also seen and was very effective.

• Be a pragmatist - I covered this one above. Agile is a tool, and like any tool; it doesn't apply to all problems. If it doesn't make sense to apply to some component, then don't apply it there. Your goal is to ship good software; don't forget it.

Answer 2

It is very possible to implement in small increments and end up with well-structured maintainable code. In theory, it's very simple: if you ensure that the code is well-structured after each change, it will always be well-structured. Your code is becoming poorly structured because you are continuing to add code when you should be refactoring.

No matter how much time you spend on design, eventually the requirements will change in some unanticipated way and you will have to write code that does not fit the original design. Then you will have to make an incremental change. If you have a good set of unit tests, and are skilled at refactoring, then you will be able to keep the code well-structured as you meet new requirements.

Answer 3

"Well-designed" is subjective

What does "well designed" mean to you? to the Product Owner? to the Customer?

Is "well designed" a goal of the product owner? a goal of the customer? or just You?

Is what you consider "not well designed" still meeting the Product Owners expectations and making the customer happy? Then that is pretty "well designed".

Good Enough and YAGNI

Nothing in most Agile methodologies speaks of "well designed", because any system that the Product Owner accepts the stories as complete and the customers believe it meets their requirements is "well designed".

It is expected that the developers are professionals and will pragmatically use best practices, appropriate designs and idioms to implement the features and stories.

If you are not factoring in the time to do things correctly that is a developer problem, if the Product Owner is demanding things in less time that these can be done, it is their prerogative to do this, and your responsibility to educate them about the consequences in the form of technical debt stories.

SCRUM

_{The Agile Methodology that can be written down, isn't the Agile Methodology." - Jarrod Roberson}

SCRUM is supposed to be a framework of tools to manage the total lifecycle of a software product. It isn't supposed to be a rigid set of things, just a good place to start and hopefully improve on.

Most shops I have worked in have what are called Sprint ZERO, Sprints for the senior members of the team to sketch out an overall architecture or theme of the product.

Stories that are larger than say 20 usually get broken down until they are actually a few 3 - 5 point Stories. One of these stories is, "As a team we need to meet to discuss how we are going to design this feature, so that we will have as little technical debt as possible given the time allotted."

Generally

In general a "well designed" system, is Good Enough and follows YAGNI.

Agile, and SCRUM in particular as an implementation of Agile are more about how to produce a product as transparently as possible to the Product Owner / Sponsors.

It isn't about anything technical design/architecture wise. It is more a philosophy about how to approach delivering software that meets the customers needs and expectations. If there isn't what you call "well designed" parts, that is not a failing per se, as it is not a fundamental part of the philosophy.

Answer 4

We've been working with scrum for several months, and I want to share my experience concerning your problem.

A few months ago I was given a big piece to implement. I had all the specifications ready, so I had to implement new features accordingly. The task was to take about 5-6 weeks (as I estimated). I spent the first week working on design only. The task was a little complicated: there was a main object which, as I concluded from the spec, had 15 different states, and the UI was to have different behaviours for every state.

I designed the whole workflow, and DB structure and classes afterwards.

I don't see any other approach in my case. If I dived into coding at once, I would have got a big nasty mess at the end _ almost impossible to maintain and extremely difficult to make any further changes. But the changes came in the next 2 weeks, so I had to rework the code. This was easy now, with the initial thought-out design. This saved our time, money and nerves.

After this experience I'm absolutely sure that its worth making an acceptable design in the beginning, than hope that with some miracle you'll have it at the end.

Answer 5

Hind-sight is 20-20. If you had the information at you disposal at the beginning of the project to figure the whole thing out and then write the code in a few weeks, I suggest you do it.

You are not giving enough credit to all of the insights you gained, the approaches that were tried and failed/had to be changed, and the increased ability of the client/users to provide requirements enough credit.

Why would anyone with a history of successful water-fall projects, switch to an agile methodology?

Answer 6

You always need the bigger picture of what the end goal should be, and what features are intended to be implemented and when, before you start a project. Working on stories as atomic tasks is asking for trouble. You need to keep future requirements in mind as much as possible.