How can agile be applied to applications involving complex processing?

Question

Most of the literature on agile seems to be biased towards CRUD type business applications where the user is pretty much aware of what is going on behind the scenes. (That's fine because most of the code being written probably belongs to this class.)

For this type of application the relationship between user stories (requirements) and development tasks is mostly straightforward: Just split up the user story into a few tasks.

But there is another type of application where most of the code has to deal with complex processing that is not directly visible to the user. Examples would be:

• Compilers
• Image analysis systems of self-driving cars
• Fluid flow simulation systems

Here it can get really difficult to relate tasks and user stories. Are there techniques to overcome this issue or is it just something we have to accept and make the best of it?

Answer 1

This turned out to be longer than I had planned (it had started as a comment), but I hope the added length/details are helpful and you find them justified.

Agile is not specific to CRUD apps

Most of the literature on agile seems to be biased towards CRUD type business applications where the user is pretty much aware of what is going on behind the scenes. (That's fine because most of the code being written probably belongs to this class.)

I think this is because it's easier to create easy-to-follow examples of this type, not really because the methodology is aimed at those kinds of systems. If you create a not-so-easy-to-follow example, you risk getting the reader stuck trying to understand the example when your point was supposed to be teaching the reader about agile concepts.

User Stories != Requirements

For this type of application the relationship between user stories (requirements) and development tasks is mostly straightforward: Just split up the user story into a few tasks.

A user story is not the same as a requirement. True, there can be some overlap depending on how 'high-level' the requirement is, but generally not the same. I get the impression that you're running into the same pitfall a lot of my former managers fell into: thinking about user stories simply as synonyms for "requirements", which is similar to when SVN users try to transition over to Git, but keep thinking in terms of SVN. (They then run into problems due to the bad starting assumptions.)

IMHO, a key difference between requirements and user stories is that requirements specify, in detail, how certain system components ought to behave; they're specifications that include inputs, outputs, assumptions/pre-conditions, possible exceptions raised, etc. They focus on what the system does.

OTOH, user stories focus on the expected outcome for the end-user without trying to create a detailed behavioral specification for system components. They focus on the expected user experience.

What I used to do, and this was a practice my team adopted, was to break down user stories into tasks. Your tasks could be as specific or vague as you wanted/needed them to be, but they were meant to be your progress indicators for actual work done towards getting the story to a done-done state.

Example

I roughly recall a US I worked on years ago where users needed to self-assign test cases so that everyone in the team was aware of which TCs they were working on to avoid duplicated efforts; the UI was a(n internal) web application. The user only saw a button, but the story was divided into several tasks that included some technical implementation details, etc.

User Visibility

But there is another type of application where most of the code has to deal with complex processing that is not directly visible to the user.

Is it possible to make it visible to the user in some way?

Consider a GPS. When you've missed your turn, you won't see the actual route re-calculation process, but the user does receive some useful feedback (e.g. "Recalculating...").

Compilers may display warnings or errors, or include new settings/options in the GUI for users to see that something new has been added. I'd think the users for compilers would be programmers, right? Wouldn't they see support for a new standard added?

While supporting a new standard would likely be at the feature level and would need to be broken down into user stories, have you made sure that, at least in some cases, you're not trying to use stories when you should be using features instead?

Image analysis in a car could be phrased in a way that lets the user know that the chances of ending up in a car crash have been reduced. For example:

As a passenger in a self-driving car, I need the probability of the vehicle causing an accident by crashing into an unrecognized object to be as close to zero as possible, so that I can travel more safely.

That US captures, at a high-level, things you'd normally have to specify using a combination of functional and non-functional requirements -including security, safety, etc.

However, a requirement might be more about the system; e.g.:

Function abc in component A must have the tolerance threshold value decreased in the image comparison algorithm to better detect objects moving slowly.

To me, this would easily be a task under the user story I mentioned above, titled something like: Decrease tolerance in function A.abc and then include other relevant details in it.

For a fluid simulation system, you could even have a progress bar that provides feedback about background tasks the system is performing, if this makes sense. (There's always a way to inform the user of something, though you may want to avoid being spammy.)

I don't know enough about the particular domains you've mentioned to come up with better and/or more realistic examples, but if there's a take-away here is that you can use different ways to provide user feedback about something less visible that the system might be doing, i.e. there might be ways to make invisible things a bit more visible. (Even if it boils down to writing a set of release notes that documents how much faster the system's performance is now due to your efforts, etc.)

Relationship between Stories and Tasks

Here it can get really difficult to relate tasks and user stories.

Our approach was to keep user stories focused on what the request was, why it was made, and what things needed to be true to consider the US "done". The how was always left out of the US and left to the developer(s).

The developer(s) would break down the problem described in the US into a set of tasks that they would work on.

I'm saying this as someone who, for the most part, did back-end server-side programming, which is probably as "invisible" as you can get for the end-user.

Depending on what I needed to do, I'd sometimes use AJAX to show a simple "loading..." animation/gif so that the user knew they needed to wait a bit for something else to complete, without getting the wrong impression. Sometimes it was as simple as this. A task for this would be appropriate.

Different Paradigm, Practicing, and Experience

Are there techniques to overcome this issue or is it just something we have to accept and make the best of it?

Beyond accepting the paradigm shift, practicing, and accrued experience, probably not much more to say. I often saw people trying to take shortcuts through the process. I'd advice against that, especially if you're getting started. As you get more experience, you can allow some flexibility, but avoid undermining yourself.

Given your prior wording, you're still thinking about stories as if they were "renamed requirements", which I think is a false assumption. I think this is a symptom of a deeper issue regarding the fundamental differences between Agile and non-Agile approaches.

Secondly, I think you should accept that agile is a paradigm shift compared to waterfall, which means that, while the process has similar goals, they go about it in very different ways. (Think SVN vs Git, if that helps.)

Try to improve your current understanding of the conceptual differences between requirements and user stories and accept they're not the same thing.

Learning from your Sprints - Retrospectives

What I cannot stress enough is the retrospective between Scrum Master and Developers at the end of each sprint. That is the place where they discuss things that "went well" or "didn't go well" in an honest/transparent way, and what doable changes will be implemented for the next sprint to address the "didn't go well" points.

That allowed us to adapt and even learn from each other's experiences, and before we knew it, we had improved significantly as measured by the general consistency of our team's velocity.

Answer 2

Agile principles can certainly be applied in these cases. How?

• Compilers can have new language features added later in separate user stories
• Image analysis systems can have new features added as different image classifications
• Fluid flow simulation systems can take different model aspects into account in their simulations

You don't have to eat the entire elephant in one bite. Agile just asks that you show you've cleaned your plate before the next serving of elephant.

Answer 3

I find that people who adhere to strictly to user stories will either just engage in a very silly exercise of coming up with far fetched ways in which back end technical changes impact the user (unbeknownst to the user of course because they're just some naive user and you're talking about complex changes in your data analysis pipe line or something of that sort) or they will just be at a complete loss for "how do we possibly organize this work!?!"

I think the obvious solution is to be more pragmatic. If the work is very technical in nature and doesn't have a particularly noticeable impact on the user, don't lose sleep trying to explain how it does. Just choose an obvious and simple way in which it can benefit users and then orient the story around the details necessary for the developers to do their jobs. I find it incredibly frustrating when a PO insists on not having technical information in story when it's absolutely necessary. It's just not a very wholistic view on what that artifact (the story) really is. Like they think it exists just for them, in most cases it's important to the engineers as well.

For most of these technical tasks there is some low hanging fruit with regard to user impact, whether that is improving efficiency so future deliveries will be speedier, improving performance, reliability ect. They're not really what people tend to think of when they think 'user stories' but if the business wants to understand why you would undertake technical debt or something to that effect, then these explanations are usually the simplest to provide.

tl;dr don't let a scrumnazi make your life extra difficult simply because they're too much of a square to adapt. Being adaptive is a core concept of agile after all. Strict adherence to Scrum or Agile usually flies in the face or pragmatism and practicality (what actually works best).

Answer 4

I think the issue is in giving user stories a meaning they do not have. Scrum uses the term PBI, or Product Backlog Item, which I think is infinitely better. PBIs will often follow a user story format, for example, you might have a PBI like "Subscribers should be able to view their subscription details", but you also might just as easily have a PBI like "Create a stored procedure to get subscriber details".

User stories are a tool. They help you form feature descriptions and requirements based on putting yourself into the shoes of a user. But, just as a wrench is useless when you need to hang a picture, there's times when you may not need a user story.

That said, many teams actually just play fast and loose with the "user" part. They may have "user stories" like "As a developer, I need to be able to call a stored procedure to get subscriber details", essentially a "developer story" as it were. This is an equally valid option, but personally, I say as long as your can describe what needs to be done and come up with a set of acceptance criteria, it hardly matters if you have an actual user story behind it or not.

Answer 5

These types of applications are exactly the ones where differing expertise is present and will develop further. The team members will have due to differing education, different hobby projects and different past work experience differing skills. Futhermore, if somebody develops a particular piece of code, the developer can be expected to be the one who knows the code best. So it might make sense to give further development tasks involving the same piece of code to the same developer.

In the most popular agile process, Scrum, there is planning poker where each task is assigned a difficulty level. The difficulty level is not dependent on the person who is doing that task according to the process. Then during the sprint, people are considered homogeneous so that each person is expected to be able to pick every single task and implement it. In simple CRUD like projects, this assumption holds. But in very complex and difficult projects, it certainly doesn't.

I wouldn't use an agile process for these kinds of projects. Your best choice is to avoid any formal processes and just use good project management. When deciding who implements a particular feature, consider who has the best skills needed for this feature and best knowledge of the existing code. No process is needed for this. You will probably want to write good design documents for all the features and keep them up to date. Note that I'm not promoting a waterfall-like model here: the design documents will not all be written at the start of the project; you will instead write new design documents as new features are needed.