Agile - What we doing wrong?

Question

I'm a developer in an agile team, and we try to use Scrum.

So I'll put here a hypothetical problem to illustrate the situation.

We have a very old app, using some messy and bad maintainability JQuery code. We also have parts of the app using React, and those parts are way easier to update/maintain. Besides that, the company goal is to make a client-single-page-app, on React, so using JQuery gets you further away from that.

When we do the planning, we always go for the easy solution in terms of development time, so for instance if we are making a new dialog or something, we use the old JQuery because its quicker, and we say that we're go back later to tidy up and transform into React, but that rarely happens.

We get the requirements of what we have to do from user stories (which are well done IMO, they are slim but they explain what we are doing, and why we are doing it).

Sometimes, the requirements of new features are very slim, so for an example if a requirement says "make a dialog that loads tons of contents" but does not says to implement a loading feature, we, in most cases, won't implement it, even though we all know it would be better for the customers, with the reason that this could compromise our sprint goal (even though I personally believe it would not).

The result is that our codebase is a big mess with very bad maintainability, and new features sometimes, are very small and take a full sprint to make (something that could be achieved in a single day in a good codebase) mainly because of this development strategy, just go fast, do the minimal.

In this case, what are we doing wrong? Should we tackle the solutions in a more complete way so we are not aways writing bad code and rewriting code we just wrote last week? Or should we keep doing that just making sure all that code is being rewritten? What would be a good agile approach to this problem?

Answer 1

This has nothing to do with Agile or Scrum.

The problem with "duct tape it now and we'll fix it later" is that later never comes and in the mean time you are accumulating a lot of technical debt.

The first step to recovery is to recognize the problem and stop making it worse.

For every new user story, the team should consider "what's the right way to code this?", not "what's the quickest way to hack this?" and plan the sprints accordingly.

To clean up the existing problem, see the excellent answers to I've inherited 200K lines of spaghetti code — what now?

Answer 2

What you have there is what Martin Fowler calls "flacid scrum".

If you properly read all 12 Principles behind the Agile Manifesto, you will find out you fail at most of them.

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

Can you say you deliver truly working software? Or just software that barely works?

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

Can you say your process is sustainable? Do you make decisions with sustainability in mind? Or do you pick solutions that solve current problem without taking into account long-term effects?

Continuous attention to technical excellence and good design enhances agility.

The truly major principle. I believe this should be put in HUGE RED LETTERS on the page. This is where you fail the most.

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

And most obviously. If you find out your behavior is not leading to desired outcomes, you should change it. If your team cannot see it has problems, then it cannot start fixing them.

From your comment

How to prevend that in agile ?

First, by learning what agile actually is. Scrum is not agile. Some would say Scrum is worst of agile frameworks, as it is too easy to reach your exact situation. You should learn about other agile frameworks. The one I would recommend is Extreme Programming. Which clearly solves your problems. The solutions are not simple (focus on technical excellence through robust automated testing, pair programming and continous delivery), but highly effective. As reported in State of DevOps report.

Answer 3

What you describe is - at least in my experience - a quite common emergent pattern of teams trying "to be Agile". It is open for debate if this is actually part of Agile itself or a common mis-implementation of it, is against the agile manifest/principles or an inherent consequence of it, and so on. Just from an empirical standpoint and just based on my own small sample set of experience (and the people I talk to), if a team is agile it seems to have a higher than average chance of running into this pattern. Let's just leave it at that and focus on your concrete example.

There are two separate aspects to what you describe:

• Missing common understanding/vision and therefore not being efficient
• How to measure success/progress and total cost

Going down the wrong path or running in circles

In my experience, the main reason for this to happen is that in an attempt to produce code quickly, teams actively push aside use cases or requirements they already know or could easily find out about. Think of it this way: 10-20 years ago, people tried to write giant specs and think of everything in advance and often failed. They either took too long or overlooked something. One of the learnings of the past is that in software development there are things that you cannot know and things change a lot, hence the idea of iterating quickly and producing some sensible output fast. Which is a very good principle. But today, we are at the other extreme: "I don't care about this because it's part of the next sprint" or "I don't file that bug, I deal with it when it comes up again". My advice would be:

1. Gather all the high level use cases, requirements, dependencies and restrictions you can find. Put it in some wiki so all stakeholders and devs can see them. Add to them when something new comes up. Talk to your shareholders & users. Use this as a check list while developing to prevent implementing things that don't contribute to the final product or are workaround/hacks that solve one problem but cause three new ones.
2. Formulate a high level concept. I'm not talking about designing interfaces or classes, but instead roughly sketch out the problem domain. What are the main elements, mechanism and interactions in the final solution? In your case, it should make it obvious when using the jquery-workaround helps as an intermediate step and when it only causes additional work.
3. Validate your concept using the list you gathered. Are there any obvious problems in it? Does it make sense? Are there more efficient ways to achieve the same user value without causing long time tech debt?

Don't overdo it. You just need something so everyone in the team (including non-devs) has a common understanding what the best path to your MVP is. Everyone should agree that there are no obvious oversights and it could actually work. This in general helps prevent going down dead ends or having to redo the same thing multiple times. Agile can help you deal better with the unexpected, it is no argument to ignore what is known.

Be aware of the sunk-cost-fallacy: if you start with one architecture or database type, most people are hesitant to change it mid-project. So it's a good idea to invest some time in having an "educated best guess" before starting to implement stuff. Devs have a tendencies to want to write code quickly. But often having a couple of mocks, live prototypes, screenshots, wireframe, etc allows for even faster iteration than writing code. Just be aware that every line of code written or even unit tests make it harder to change your overall concept again.

Measuring Success

A completely separate aspect is how you measure progress. Let's say the goal of your project is to build a tower that is 1m high using things lying around. Building a house of cards can be a totally valid solution if for example time to market is more important than stability. If your goal is to build something that lasts, using Lego would have been better. The point is: what is considered a hack and what an elegant solution dependents entirely on how the project's success is measured.

Your example of the "loading" is pretty good. I had things like this in the past where everyone (including sales, PO, users) agreed it was annoying. But it had no impact on the success of the product and caused no long term debt. So we dropped it because there were more valuable things to do with the dev-resources.

My advice here is:

1. Keep everything, even small bugs, as tickets in your ticket system. Make an active decision what is within the project scope and what not. Create milestones or otherwise filter your backlog so you always have a "complete" list of everything that still needs to be done.
2. Have a strict order of importance and clear cut off point where the project could be considered a success. What level of stability / code quality / documentation does the final product actually need? Try to spend every day of work as best as possible by picking from the top. When working on one ticket, try to solve it completely without introducing new tickets (unless it makes sense to post-pone things due to lower priority). Every commit should bring you forwards towards your end goal, not sideways or backwards. But to stress it again: sometimes a hack that produces additional work later on can still be a net positive for the project!
3. Use your PO/users to figure out the user value but also have your devs figure out the tech cost. Non-devs typically cannot judge what the true longterm cost (not just implementation cost!) is, so help them. Be aware of the boiling-frog problem: lots of little, irrelevant problems can over time bring a team to a hold. Try to quantify how efficient your team can work.
4. Keep an eye on the overall goal/costs. Instead of thinking from sprint to sprint, rather keep a mindset of "can we as a team do everything needed until the end of the project". Sprints are just a way to break things down and have check-points.
5. Instead of wanting to show something early, plot your course on the fastest path to a minimum viable product that can be given to the user. Still, your overall strategy should allow for verifiable results in between.

So when someone does something that does not fit into your final implementation goal, ideally don't consider the story done. If it is beneficial to close the story (e.g. to get feedback from customers), immediately open a new story/bug to address the short comings. Make it transparent that taking shortcuts does not reduce costs, it just hides or delays them!

The trick here is to argue with total cost of the project: if for example a PO pushes for taking shortcuts to make a deadline, quantify the amount of work that has to be done afterwards to consider the project done!

Also beware of criteria-based-optimisation: if your team is measured by the number of stories they can show at a sprint review, the best way to achieve a good "score" is to cut every story into ten tiny ones. If it is measured by the number of unit tests written, it will tend to write lots of unnecessary ones. Don't count stories, rather have a measure of how much of the needed user functionality works, how big the cost by tech debt to be solved within the project scope is, etc.

Summary

To boil it down: Going fast and minimal is a good approach. The problem is in interpreting "fast" and "minimal". One should always consider the long term cost (unless you have a project where this is irrelevant). Using a shortcut that only takes 1 day but produces tech debt of 1 month after the shipping date costs your company more than a solution that took 1 week. Immediately start writing tests seems fast, but not if your concept is flawed and they cement a wrong approach.

And keep in mind what "long term" means in your case: I know more than one company that went bust by trying to write great code and therefore shipped too late. A good architecture or clean code - from a company perspective - is only valuable if the cost to achieving it is less than the cost of not having it.

Hope that helps!

Answer 4

Strictly from a scrum perspective, it sounds like what you're doing wrong is that you're not working with the client. You need to work together with the client to come to an understanding of what they need and not just what they want. Do they need a series of quick fixes, or do they need a stable, maintainable system that will serve them in the long term? That can be hard to determine, but quality is as much of a requirement as a background color or performance benchmark. The customer needs to be aware that stability and maintainability aren't free, and has to be engineered into the product.

If they say it is the former, you are doing nothing wrong -- assuming you're explaining to them in the sprint reviews that you're cutting engineering corners to meet their goals.

If they say it is the latter, then what you're doing wrong is that you're not giving them what they want.

One of the cornerstones of Scrum is transparency. If you're doing scrum, you should be doing sprint reviews with the customer. In those reviews, are you telling the customer that you're cutting corners in order to deliver software faster? If not, you should be. You need to be 100% clear with your customer about the ramifications of your design choices, to give them a chance to make an informed decision as to whether you are delivering your software with an appropriate level of quality.

Answer 5

Ewan is right. The reason management likes scrum is because they get to demand features in staccato style and get results quickly. Until the resulting mess is someone elses problem.

Now that I have your attention, please let me explain. It is not Scrum as such. It is the typical setting of a strong product manager and a weak development team that is unable to put down reasonable and realistic estimates because they feel the pressure. So they come up with far to optimistic estimates and get themselves deeper into trouble, cutting corners in order to deliver in time.

In scrum you (as a developer) get to do your own planning. No one is telling you to deliver some feature in x days. If someone is telling you to deliver in x days, you are not doing Scrum.

Whatever the problem is that needs to be fixed, claim your time. Do you think you need time to rework something first? Incorporate it in your estimate. Can you afford to do that?

Answer 6

Let's examine what you are doing, setting aside Agile for a moment.

When we do the planning, we aways go for the easy solution in terms of development time, so for instance if we are making a new dialog or something, we use the old jquery because its quicker, and we say we gonna go back later to tidy up and transform into react, but that rarely happens.

This is called "Taking Technical Debt". Martin Fowler described the "Quadrant of Technical debt" in a blogpost of his along the two axes: "Reckless vs. Prudent" and "Deliberate vs. Inadvertent".

You explicitly decide to use the known old technology jquery that gets you further away from one of your express goals (namely a single-page app). You do this to deliver "quickly". This is Deliberate.

What this calculation of "quickly" does not include is the time you need to implement the functionality in react afterwards. You choose an alternative that has only downsides over the alternative you know to be the correct one (namely taking the time to implement the feature in react) based on an assessment that speed is of the essence. This is Reckless.

Martin Fowler sums this kind of debt under "We don't have time for design". That is an appropriate choice in an environment where you're not expecting to maintain the code or even expecting to code for more than a few days. But your project is a long-running project that explicitly involves maintenance for your customer(s)

What you're doing is wrong on the very basic level. It's bad engineering!

You took on technical debt, ignoring that this debt needs to be repaid and charges interest. And you kept doing that until the interest rate on your debt started to close in on your the available work during your sprint.

What you should be doing is reducing the level of debt. Talk to your boss, talk to your client. You need to be working on maintainability yesterday.

Answer 7

You are not doing anything wrong. This kind of methodology is designed to deliver features to spec and as fast as possible.

If you do have secondary goals that you are working towards then its best to express them as 'non-functional requirements' or a 'definition of done'.

for example, you could have a non-functional requirement:

"All new features must be written in React"

and

"All asynchronous calls must implement a loading spinner and error handling"

You just have to get your Product Owner (or equivalent) to agree that these are things that are worth doing, rather than sneaking them in because the developers like them.

Answer 8

Just stop using Agile...

Or rather, stop trying to do something a certain way purely because that is what (your understanding of) agile (or scrum etc...) dictates. Trying to apply one (mis)interpretation of one of these terms to a project at the wrong stage can quickly become the worst course of action. Use your reason instead.

The reason your project, and almost every other project in the world, is a mess of code and divergent approaches, is due to lack of centralised, omniscient architectural design (there, I said it).

The reasons this might be lacking are:

• The architect doesn't have the expertise (Like your first ten hobby projects)
• The architect doesn't have the time
• The architect doesn't have the power (Manager says no, or yes but only for some parts)
• The team has faith in some voodoo methodology that will save them (It will all iron itself out because we're using Agile)

The simple solution is to drop all these magic words, and look at the reality of the situation, which can be summarised as:

1. The state of the code is impeding the team's ability to deliver on time and bug free.
2. The more features we add, the worse it will become.
3. Therefore it really makes sense to pause, reassess, and (perhaps drastically) redesign parts.

You will naturally come to ask why it got to this state in the first place, with the finger of blame spinning round and round. The answer is that this is inevitable: as your design matures you realise you should have done it differently, but you couldn't have foreseen it. Furthermore this is not a once-per-project realisation, it will happen several times, and you need to plan for it.

Having said that, there are plenty things managers can do to exacerbate things:

1. Deathmarching your devs to deadlines.
2. Stating that devs may only log time against tickets, without there being tickets for "thinking, consolidation and quality refactoring" and a generous time allowance on those.
3. Not giving any one person ownership of the architecture for long enough to get a handle on it
4. Not allowing that person to make the changes they think are needed

Looking at it this way, it's easy to see how some interpretations of agile & scrum will actually march you down this route even quicker!

One approach is to create tickets for each bit of refactoring. The problem is that you often don't realise you need a big refactor until you start working on a smaller ticket, which pushes deadlines back, and it the ticket goes through approval loops it just slows everything down.

Another approach is to plan sprints to use only 25-50% of your team's capacity. Devs then log their time on the real tickets (log the time it should have taken without refactoring) and refactoring time (one big ticket for the week, no approval loops, only discussion between devs). If there is no refactoring, you can pull tickets from next week's sprint. You adjust the percentage slider for coming weeks as the project's underlying code improves.

So to answer "what are we doing wrong" I'd say you are placing trust in a methodology over common sense. You even ask for an "agile approach to this problem". I'd say drop the words and think about the actual problem. If you then really want to pick apart various manifestos trying to decipher whether your final common sense approach does indeed fall under the guise of "agile" or "scrum", by all means go for it :-)

Answer 9

You are doing nothing wrong. These kinds of situations occur frequently in agile development in general and scrum in particular, e.g:

Team member: This whole project is sinking like the Titanic while the band is still playing. We have to do something.

Management: It is because YOU are not adhering to the methodology!

... and problem solved, or for example:

Team member: OK, but who wrote this code? I can't make any sense of it. Worse yet, it is only partially working but we are relying on it heavily for core features.

Scrum master: Oh, he left about six months ago. When will you be able to get this finished?

Team member: I really don't know. We might have to rewrite the whole thing.

Scrum master: That is unacceptable, I need commitment from you. Plus a rewrite will never be authorized by the management. We need to add value to the product for the customer.

Team member: I don't think you understand the amount of "technical debt" that is already there.

Scrum master: It's you who don't understand how Scrum works.

:)

But seriously, if the "product owner" does not agree to the necessary changes, then there is nothing that you can do. It's that simple. The "product owner" prioritizes what needs to be done next. Unfortunately, he is not a technical person. Oftentimes, neither is the "scrum master". If they both balk at what needs to be done, then that's it; that too is often the case.