Iteratively improve software architecture & quality in an agile process?

Question

Or to put it another way how to ensure that architecture or quality doesn't suffer, doing agile.

Some of the understandings in handling architecture in agile are below(generally applies to testing as well)

* Do architecture until risks fall off your radar * minimal design up front, so that you will not pay a horrible price for reasonable change requests. * "just enough" architecture/infrastructure per each feature/requirement

(The above were found in searching other related StackOverflow questions)

---

A hypothetical example/scenario is that there is a requirement to develop a order processing module in the existing system. The below guidelines are negotiated and agreed upon.

1. The architecture for module should be able to handle processing of 10 orders in the span of a minute.
2. Time to market is 4 weeks, planning is done to handle 2 sprints of 2 weeks.
3. The Business is satisfied with average performance expectations.

Once the module is delivered, lets say the following improvements/changes are requested.

• Improve the processing to 15 orders per minute.
• Improve performance by 25%.
• Time to market is 2 weeks.

So assuming the further requirements from the customer/business come and which are welcome, but in future as the iterations go on, the architecture needs to be overhauled, and testing the quality of the module also becomes more time consuming.

---

And the above is not a happy result, as estimations increase, and there is an understandable inability to meet further expectations or requirements quickly. Even if it can be done in the given timeframe with sufficient quality, just wish that we could have done something about it well before and not face this situation at this point of time.

In hindsight, architecting to scale in the first iteration would be ideal, but that was not the goal and focus. The focus is always on Time to market, and on acheiving on meeting the business/customers expectations in a given timeframe. Unfortunately, in the long run it does take a toll on development.

What can be done to avoid the mentioned pitfall in the example, in improving the architecture or testing process iteratively?

Answer 1

I think that mostly you are mixing the wrong levels of abstraction and thus, you are getting stuck.

First of all, there can be requirements to change the architecture, but they do not normally make good stories. Say for example you need to improve performance by x%. This is not a user story and it might not be achievable at all. There are systems that can't be improved, there are systems that can only be improved at very great cost, etc.

The first two missing bits that come to mind are:

• What is the business value of improving performance by x%?
• What are the correct boundaries in which we operate?

Can't we simply purchase a x% faster server, for example?

There are no "architectural" stories in agile development because architecture has no actual value to the user.

Architecture is a tool, it has practical value and it is useful in agile development, but it has nothing to do with architectural stories. Architecture is an activity, like design, refactoring, debugging, etc. None of these activities would ever make a good story, but all of these activities are typically part of the development of a story.

Another point to keep in mind, architectural qualities become part of the "definition of done". If you have a requirement that every web page should be delivered in less than 500ms, making this part of the DoD ensures that this applies to every story, and it becomes a criterion for its acceptance testing.

Furthermore, changing significantly the performance level set as "definition of done", is generally a risky move. Unless you simply made a mistake and set the wrong value, changing the performance profile of an application is difficult on many levels -- and it's especially difficult to estimate.

As an alternative you can limit the impact by specifying in which cases you want this change and why. This allows the team to be set for success, with results which are likely to be achievable. Nobody can be sure that you can change the performance of a complex application on a general level, however estimating a single page or feature is more feasible.

Regarding hardening stories: the only approach that I've seen work (and what we use at Stack Exchange) is having one person on bug duty every iteration. This ensures that there's always someone on the piece when problems arise, is fair in nature, and completely sidesteps the scheduling problems that arise when treating bugs as stories ("there's a new bug in the middle of an iteration, do we fix that or close the stories?").

Answer 2

Agile architecture is a running gunbattle. I try to implement it in a just-in-time methodology, but ultimately, it is in direct opposition to agile methodologies. That doesn't mean it's bad. It is very important, it's just not something that fits into the agile process.

Up front architecture and design is really a hedge against the risks of agile development. You have to balance your risk mitigation with your time to market. In this case, it sounds like you came down a little on the wrong side of the risk/reward equation. Next time you might do too much architecture, and waste a bunch of time. Unfortunately, it just takes experience and wisdom to figure out how much is "too much" architecture for an agile process.

Answer 3

It's a common question - and one that a lot of our customers ask when we discuss agile.

The key word in your question is "hindsight". For every time that I've encountered an "if only we had known" moment, I've encountered many "You Ain't Gonna Need It" moments (YAGNI). Trying to fix all current and future problems/requirements in a single release will drive you crazy, make the software complex, late and expensive, and generally slow you down.

On the whole, the rework required to improve performance for an existing, well-crafted feature is much less wasteful than investing in "architectural" solutions to hypothetical problems.

When explaining this to my clients and my teams, I generally say that the team's job is to optimize throughput and quality; the client's job is to make sure we're always working on the most valuable tasks. The whole point of agile is that "valuable" changes over time, and that the team responds to this quickly. As long as you achieve that, rework isn't a big deal, even if it feels inefficient.