22

I'm a developer in an agile team, and we try to use Scrum.

So I'll put here a hypothetical problem to illustrate the situation.

We have a very old app, using some messy and bad maintainability JQuery code. We also have parts of the app using React, and those parts are way easier to update/maintain. Besides that, the company goal is to make a client-single-page-app, on React, so using JQuery gets you further away from that.

When we do the planning, we always go for the easy solution in terms of development time, so for instance if we are making a new dialog or something, we use the old JQuery because its quicker, and we say that we're go back later to tidy up and transform into React, but that rarely happens.

We get the requirements of what we have to do from user stories (which are well done IMO, they are slim but they explain what we are doing, and why we are doing it).

Sometimes, the requirements of new features are very slim, so for an example if a requirement says "make a dialog that loads tons of contents" but does not says to implement a loading feature, we, in most cases, won't implement it, even though we all know it would be better for the customers, with the reason that this could compromise our sprint goal (even though I personally believe it would not).

The result is that our codebase is a big mess with very bad maintainability, and new features sometimes, are very small and take a full sprint to make (something that could be achieved in a single day in a good codebase) mainly because of this development strategy, just go fast, do the minimal.

In this case, what are we doing wrong? Should we tackle the solutions in a more complete way so we are not aways writing bad code and rewriting code we just wrote last week? Or should we keep doing that just making sure all that code is being rewritten? What would be a good agile approach to this problem?

  • 21
    "The result is that our codebase is a big mess with very bad maintainability, mainly because of this development strategy, just go fast, do the minimal." - It sounds like you already have a good idea of the problem, but I'm not sure it really has much to do with Agile. You can get duct tape coding no matter what methodology you use. – Nathanael Jul 6 '18 at 15:40
  • How to prevend that in agile ? People understand the incremental as duck taping then fixing. – Gabriel Slomka Jul 6 '18 at 15:42
  • 7
    "People understand the incremental as duck taping then fixing." - that's certainly not what scrum is. If "people" think that, they misunderstand scrum. – Bryan Oakley Jul 6 '18 at 16:37
  • 9
    To cite Eric Lippert: if you dug yourself into a hole, first thing to get out is: stop digging. – Doc Brown Jul 6 '18 at 16:54
  • 5
    Does your team follow the "boy scout rule" (leave a place always in a better state than it was when you entered it)? Start with that. Moreover, codereviews, writing tests and regular refactoring are also helpful techniques. – Doc Brown Jul 6 '18 at 16:59
57

This has nothing to do with Agile or Scrum.

The problem with "duct tape it now and we'll fix it later" is that later never comes and in the mean time you are accumulating a lot of technical debt.

The first step to recovery is to recognize the problem and stop making it worse.

For every new user story, the team should consider "what's the right way to code this?", not "what's the quickest way to hack this?" and plan the sprints accordingly.

To clean up the existing problem, see the excellent answers to I've inherited 200K lines of spaghetti code — what now?

  • In addition, I feel most problems like this are caused by not having an experienced manager who knows how to work out these problems and, instead, replace the manager with named methodologies one reads about online. One advantage of this is now the method gets the blame instead of the manager. – Rob Jul 7 '18 at 10:50
  • 1
    The answer is simply this. Well put and very precise. SCRUM is just a way to work, if you decide to work with duct tape instead of finishing tape, that's on you. – coteyr Jul 7 '18 at 12:09
  • You get what you incentivize for. If you keep people under constant deadline pressure (Scrum's sprints), you're incentivizing people to take shortcuts. Thus tech debt accumulates. – Michael B Aug 20 '18 at 22:21
22

What you have there is what Martin Fowler calls "flacid scrum".

If you properly read all 12 Principles behind the Agile Manifesto, you will find out you fail at most of them.

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

Can you say you deliver truly working software? Or just software that barely works?

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

Can you say your process is sustainable? Do you make decisions with sustainability in mind? Or do you pick solutions that solve current problem without taking into account long-term effects?

Continuous attention to technical excellence and good design enhances agility.

The truly major principle. I believe this should be put in HUGE RED LETTERS on the page. This is where you fail the most.

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

And most obviously. If you find out your behavior is not leading to desired outcomes, you should change it. If your team cannot see it has problems, then it cannot start fixing them.

From your comment

How to prevend that in agile ?

First, by learning what agile actually is. Scrum is not agile. Some would say Scrum is worst of agile frameworks, as it is too easy to reach your exact situation. You should learn about other agile frameworks. The one I would recommend is Extreme Programming. Which clearly solves your problems. The solutions are not simple (focus on technical excellence through robust automated testing, pair programming and continous delivery), but highly effective. As reported in State of DevOps report.

  • 6
    "Some would say Scrum ... is too easy to reach your exact situation.". I don't think that's true at all. Doing scrum wrong might lead to this exact situation, but scrum itself doesn't support the cheapest possible solution, unless that's exactly what the customer wants. – Bryan Oakley Jul 6 '18 at 16:36
  • 1
    @BryanOakley What I want to say is that if process does not prescribe doing X, then people won't do X. And Scrum does not prescribe any practices that would reduce technical debt. Quite the contrary, as if only work to be done is defined by PO, then no technical debt will be removed. As PO has no reason to care about it. Technical debt is only team's responsibility. – Euphoric Jul 6 '18 at 16:38
  • 2
    "Scrum does not prescribe any practices that would reduce technical debt." - nor does it prescribe any practices that increase technical debt. – Bryan Oakley Jul 6 '18 at 16:39
  • 2
    @BryanOakley The point of technical debt is that it is natural state that it increases. And work must be done to decrease it. Left alone, it will grow uncontrolably. – Euphoric Jul 6 '18 at 16:41
  • 4
    If the PO is the only one who gets input on what goes into the sprint, the PO is performing their role poorly. It's their job to decide what is most important by talking with everyone whose involved with the production process, and that includes the rest of their team. – Erik Jul 7 '18 at 5:54
9

What you describe is - at least in my experience - a quite common emergent pattern of teams trying "to be Agile". It is open for debate if this is actually part of Agile itself or a common mis-implementation of it, is against the agile manifest/principles or an inherent consequence of it, and so on. Just from an empirical standpoint and just based on my own small sample set of experience (and the people I talk to), if a team is agile it seems to have a higher than average chance of running into this pattern. Let's just leave it at that and focus on your concrete example.

There are two separate aspects to what you describe:

  • Missing common understanding/vision and therefore not being efficient
  • How to measure success/progress and total cost

Going down the wrong path or running in circles

In my experience, the main reason for this to happen is that in an attempt to produce code quickly, teams actively push aside use cases or requirements they already know or could easily find out about. Think of it this way: 10-20 years ago, people tried to write giant specs and think of everything in advance and often failed. They either took too long or overlooked something. One of the learnings of the past is that in software development there are things that you cannot know and things change a lot, hence the idea of iterating quickly and producing some sensible output fast. Which is a very good principle. But today, we are at the other extreme: "I don't care about this because it's part of the next sprint" or "I don't file that bug, I deal with it when it comes up again". My advice would be:

  1. Gather all the high level use cases, requirements, dependencies and restrictions you can find. Put it in some wiki so all stakeholders and devs can see them. Add to them when something new comes up. Talk to your shareholders & users. Use this as a check list while developing to prevent implementing things that don't contribute to the final product or are workaround/hacks that solve one problem but cause three new ones.
  2. Formulate a high level concept. I'm not talking about designing interfaces or classes, but instead roughly sketch out the problem domain. What are the main elements, mechanism and interactions in the final solution? In your case, it should make it obvious when using the jquery-workaround helps as an intermediate step and when it only causes additional work.
  3. Validate your concept using the list you gathered. Are there any obvious problems in it? Does it make sense? Are there more efficient ways to achieve the same user value without causing long time tech debt?

Don't overdo it. You just need something so everyone in the team (including non-devs) has a common understanding what the best path to your MVP is. Everyone should agree that there are no obvious oversights and it could actually work. This in general helps prevent going down dead ends or having to redo the same thing multiple times. Agile can help you deal better with the unexpected, it is no argument to ignore what is known.

Be aware of the sunk-cost-fallacy: if you start with one architecture or database type, most people are hesitant to change it mid-project. So it's a good idea to invest some time in having an "educated best guess" before starting to implement stuff. Devs have a tendencies to want to write code quickly. But often having a couple of mocks, live prototypes, screenshots, wireframe, etc allows for even faster iteration than writing code. Just be aware that every line of code written or even unit tests make it harder to change your overall concept again.

Measuring Success

A completely separate aspect is how you measure progress. Let's say the goal of your project is to build a tower that is 1m high using things lying around. Building a house of cards can be a totally valid solution if for example time to market is more important than stability. If your goal is to build something that lasts, using Lego would have been better. The point is: what is considered a hack and what an elegant solution dependents entirely on how the project's success is measured.

Your example of the "loading" is pretty good. I had things like this in the past where everyone (including sales, PO, users) agreed it was annoying. But it had no impact on the success of the product and caused no long term debt. So we dropped it because there were more valuable things to do with the dev-resources.

My advice here is:

  1. Keep everything, even small bugs, as tickets in your ticket system. Make an active decision what is within the project scope and what not. Create milestones or otherwise filter your backlog so you always have a "complete" list of everything that still needs to be done.
  2. Have a strict order of importance and clear cut off point where the project could be considered a success. What level of stability / code quality / documentation does the final product actually need? Try to spend every day of work as best as possible by picking from the top. When working on one ticket, try to solve it completely without introducing new tickets (unless it makes sense to post-pone things due to lower priority). Every commit should bring you forwards towards your end goal, not sideways or backwards. But to stress it again: sometimes a hack that produces additional work later on can still be a net positive for the project!
  3. Use your PO/users to figure out the user value but also have your devs figure out the tech cost. Non-devs typically cannot judge what the true longterm cost (not just implementation cost!) is, so help them. Be aware of the boiling-frog problem: lots of little, irrelevant problems can over time bring a team to a hold. Try to quantify how efficient your team can work.
  4. Keep an eye on the overall goal/costs. Instead of thinking from sprint to sprint, rather keep a mindset of "can we as a team do everything needed until the end of the project". Sprints are just a way to break things down and have check-points.
  5. Instead of wanting to show something early, plot your course on the fastest path to a minimum viable product that can be given to the user. Still, your overall strategy should allow for verifiable results in between.

So when someone does something that does not fit into your final implementation goal, ideally don't consider the story done. If it is beneficial to close the story (e.g. to get feedback from customers), immediately open a new story/bug to address the short comings. Make it transparent that taking shortcuts does not reduce costs, it just hides or delays them!

The trick here is to argue with total cost of the project: if for example a PO pushes for taking shortcuts to make a deadline, quantify the amount of work that has to be done afterwards to consider the project done!

Also beware of criteria-based-optimisation: if your team is measured by the number of stories they can show at a sprint review, the best way to achieve a good "score" is to cut every story into ten tiny ones. If it is measured by the number of unit tests written, it will tend to write lots of unnecessary ones. Don't count stories, rather have a measure of how much of the needed user functionality works, how big the cost by tech debt to be solved within the project scope is, etc.

Summary

To boil it down: Going fast and minimal is a good approach. The problem is in interpreting "fast" and "minimal". One should always consider the long term cost (unless you have a project where this is irrelevant). Using a shortcut that only takes 1 day but produces tech debt of 1 month after the shipping date costs your company more than a solution that took 1 week. Immediately start writing tests seems fast, but not if your concept is flawed and they cement a wrong approach.

And keep in mind what "long term" means in your case: I know more than one company that went bust by trying to write great code and therefore shipped too late. A good architecture or clean code - from a company perspective - is only valuable if the cost to achieving it is less than the cost of not having it.

Hope that helps!

  • "Think of it this way: 10-20 years ago, people tried to write giant specs and think of everything in advance and often failed.": Been in the business since the nineties and, well, no, we did not work like that. To say this is just a marketing commonplace to contrast agile to a mythical past in which people were getting it wrong by planning too much. Not to plan too much and to produce an early prototype were among the first lessons I learned back around 1998 or so. The agile movement is partly just using new words for well-known practices and marketing them as new. – Giorgio Jul 8 '18 at 12:08
  • Granted, it of course depends on one's own experiences. I actually was on a few projects with big, conservative car manufacturers and you would not believe how detailed the specs were before a single line of code was written. As much as what I described was an extreme, there are quite a few companies nowadays that don't do any proper inception (which I never experienced back in the days). There are and always were examples of every point on the spectrum between those two extreme. But I at least see that the general tendency has changed quite noticeably towards the "no inception" end. – AlexK Jul 8 '18 at 12:27
7

Strictly from a scrum perspective, it sounds like what you're doing wrong is that you're not working with the client. You need to work together with the client to come to an understanding of what they need and not just what they want. Do they need a series of quick fixes, or do they need a stable, maintainable system that will serve them in the long term? That can be hard to determine, but quality is as much of a requirement as a background color or performance benchmark. The customer needs to be aware that stability and maintainability aren't free, and has to be engineered into the product.

If they say it is the former, you are doing nothing wrong -- assuming you're explaining to them in the sprint reviews that you're cutting engineering corners to meet their goals.

If they say it is the latter, then what you're doing wrong is that you're not giving them what they want.

One of the cornerstones of Scrum is transparency. If you're doing scrum, you should be doing sprint reviews with the customer. In those reviews, are you telling the customer that you're cutting corners in order to deliver software faster? If not, you should be. You need to be 100% clear with your customer about the ramifications of your design choices, to give them a chance to make an informed decision as to whether you are delivering your software with an appropriate level of quality.

  • 3
    When working with the client, make sure you figure out what they need, not what they say they want. Pretty much any client will pick the cheapest and fastest solution to every problem, it's the engineering team's job to figure out what the cheapest option is that still covers all the things they really need. – Erik Jul 7 '18 at 5:56
  • 1
    @Erik: excellent comment. That's why I originally wrote _"to come to an understanding of what they need" rather than "... they want". I can see, however, that that's not emphasized much. I'll add a bit more emphasis and explanation. Thanks for the comment. – Bryan Oakley Jul 7 '18 at 16:53
5

Ewan is right. The reason management likes scrum is because they get to demand features in staccato style and get results quickly. Until the resulting mess is someone elses problem.

Now that I have your attention, please let me explain. It is not Scrum as such. It is the typical setting of a strong product manager and a weak development team that is unable to put down reasonable and realistic estimates because they feel the pressure. So they come up with far to optimistic estimates and get themselves deeper into trouble, cutting corners in order to deliver in time.

In scrum you (as a developer) get to do your own planning. No one is telling you to deliver some feature in x days. If someone is telling you to deliver in x days, you are not doing Scrum.

Whatever the problem is that needs to be fixed, claim your time. Do you think you need time to rework something first? Incorporate it in your estimate. Can you afford to do that?

3

Let's examine what you are doing, setting aside Agile for a moment.

When we do the planning, we aways go for the easy solution in terms of development time, so for instance if we are making a new dialog or something, we use the old jquery because its quicker, and we say we gonna go back later to tidy up and transform into react, but that rarely happens.

This is called "Taking Technical Debt". Martin Fowler described the "Quadrant of Technical debt" in a blogpost of his along the two axes: "Reckless vs. Prudent" and "Deliberate vs. Inadvertent".

You explicitly decide to use the known old technology jquery that gets you further away from one of your express goals (namely a single-page app). You do this to deliver "quickly". This is Deliberate.

What this calculation of "quickly" does not include is the time you need to implement the functionality in react afterwards. You choose an alternative that has only downsides over the alternative you know to be the correct one (namely taking the time to implement the feature in react) based on an assessment that speed is of the essence. This is Reckless.

Martin Fowler sums this kind of debt under "We don't have time for design". That is an appropriate choice in an environment where you're not expecting to maintain the code or even expecting to code for more than a few days. But your project is a long-running project that explicitly involves maintenance for your customer(s)

What you're doing is wrong on the very basic level. It's bad engineering!

You took on technical debt, ignoring that this debt needs to be repaid and charges interest. And you kept doing that until the interest rate on your debt started to close in on your the available work during your sprint.

What you should be doing is reducing the level of debt. Talk to your boss, talk to your client. You need to be working on maintainability yesterday.

2

Just stop using Agile...

Or rather, stop trying to do something a certain way purely because that is what (your understanding of) agile (or scrum etc...) dictates. Trying to apply one (mis)interpretation of one of these terms to a project at the wrong stage can quickly become the worst course of action. Use your reason instead.

The reason your project, and almost every other project in the world, is a mess of code and divergent approaches, is due to lack of centralised, omniscient architectural design (there, I said it).

The reasons this might be lacking are:

  • The architect doesn't have the expertise (Like your first ten hobby projects)
  • The architect doesn't have the time
  • The architect doesn't have the power (Manager says no, or yes but only for some parts)
  • The team has faith in some voodoo methodology that will save them (It will all iron itself out because we're using Agile)

The simple solution is to drop all these magic words, and look at the reality of the situation, which can be summarised as:

  1. The state of the code is impeding the team's ability to deliver on time and bug free.
  2. The more features we add, the worse it will become.
  3. Therefore it really makes sense to pause, reassess, and (perhaps drastically) redesign parts.

You will naturally come to ask why it got to this state in the first place, with the finger of blame spinning round and round. The answer is that this is inevitable: as your design matures you realise you should have done it differently, but you couldn't have foreseen it. Furthermore this is not a once-per-project realisation, it will happen several times, and you need to plan for it.

Having said that, there are plenty things managers can do to exacerbate things:

  1. Deathmarching your devs to deadlines.
  2. Stating that devs may only log time against tickets, without there being tickets for "thinking, consolidation and quality refactoring" and a generous time allowance on those.
  3. Not giving any one person ownership of the architecture for long enough to get a handle on it
  4. Not allowing that person to make the changes they think are needed

Looking at it this way, it's easy to see how some interpretations of agile & scrum will actually march you down this route even quicker!

One approach is to create tickets for each bit of refactoring. The problem is that you often don't realise you need a big refactor until you start working on a smaller ticket, which pushes deadlines back, and it the ticket goes through approval loops it just slows everything down.

Another approach is to plan sprints to use only 25-50% of your team's capacity. Devs then log their time on the real tickets (log the time it should have taken without refactoring) and refactoring time (one big ticket for the week, no approval loops, only discussion between devs). If there is no refactoring, you can pull tickets from next week's sprint. You adjust the percentage slider for coming weeks as the project's underlying code improves.

So to answer "what are we doing wrong" I'd say you are placing trust in a methodology over common sense. You even ask for an "agile approach to this problem". I'd say drop the words and think about the actual problem. If you then really want to pick apart various manifestos trying to decipher whether your final common sense approach does indeed fall under the guise of "agile" or "scrum", by all means go for it :-)

-1

You are not doing anything wrong. This kind of methodology is designed to deliver features to spec and as fast as possible.

If you do have secondary goals that you are working towards then its best to express them as 'non-functional requirements' or a 'definition of done'.

for example, you could have a non-functional requirement:

"All new features must be written in React"

and

"All asynchronous calls must implement a loading spinner and error handling"

You just have to get your Product Owner (or equivalent) to agree that these are things that are worth doing, rather than sneaking them in because the developers like them.

  • " This kind of methodology is designed to deliver features to spec and as fast as possible." - that is definitely not the goal of Scrum. The way you phrased it, it's not clear if that's what you meant or not. – Bryan Oakley Jul 6 '18 at 16:33
  • sorry, i guess its about delivering features over engineered and late? – Ewan Jul 6 '18 at 16:40
  • No, not really. Scrum is about working with the customer to deliver high quality software in a highly visible, iterative fashion. Scrum says nothing about delivering low quality features instead of doing proper engineering. – Bryan Oakley Jul 6 '18 at 16:45
  • 2
    if you'll forgive me a criticism you seem have a very firm idea about what scrum is about. But if I check the guide and other 'official' statements today it all seems very wishy washy. I think you would be hard-pressed to find a statement that makes a clear statement on the matter – Ewan Jul 6 '18 at 17:03
  • 1
    @Erik they think its a mess because they want to use react. The Dev Team cant just decide to refactor everything on their own. The customer would refuse to pay for the sprint. – Ewan Jul 7 '18 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.