How to manage depolarization of completed work on Agile Scrum

Question

Description is too long, but this is a pain point. I just want to understand whether this is also practice of Agile, if yes, how to overcome some of the issues mentioned below. Thank you.

We are using agile scrum for our projects and we have 2 weeks sprints. At each sprint we are delivering "something". We have huge amount of micro services with multiple dev teams work on each. Therefore we have lot of internal integrations and dependencies when there are changes. Normally a CR getting added as an Epic, each epic can have multiple development tasks for each impacted micro service.

When it comes to sub tasks, sometimes there can be tasks which takes more than 1 sprint. Such CRs/Epics normally get delivered in the next sprint or whenever they are ready.

Assume we have following tasks. (Keep focusing on MicroService03)

First Sprint

Epic 1
  - Task E1T1 --> MicroService01 -> DevTeam01  -> Estimate: 1 week
  - Tast E1T2 --> MicroService02 -> DevTeam02  -> Estimate: 2 weeks
  - Task E1T3 --> MicroService03 -> DevTeam03  -> Estimate: 2 days

Epic 2
  - Task E2T1 --> MicroService03 -> DevTeam03  -> Estimate: 1 day
  - Task E2T2 --> MicroService04 -> DevTeam04  -> Estimate: 2 weeks

Epic 3
  - Task E3T1 --> MicroService03 -> DevTeam03  -> Estimate: 1 day

Epic 4
  - Task E4T1 --> MicroService05 -> DevTeam03  -> Estimate: 1 day

Assume Epic 1, 2, 3 and 4 are ordered by priorities. Which means Epic 1 is the highest important one for the end users/customers.

Status at the end of the sprint,

Epic 1
  - Task E1T1 --> MicroService01 -> DevTeam01  -> Estimate: 1 week  - COMPLETED
  - Tast E1T2 --> MicroService02 -> DevTeam02  -> Estimate: 2 weeks - WIP 
  - Task E1T3 --> MicroService03 -> DevTeam03  -> Estimate: 2 days  - COMPLETED

Epic 2
  - Task E2T1 --> MicroService03 -> DevTeam03  -> Estimate: 1 day   - COMPLETED
  - Task E2T2 --> MicroService04 -> DevTeam04  -> Estimate: 2 weeks - WIP 

Epic 3
  - Task E3T1 --> MicroService03 -> DevTeam03  -> Estimate: 1 day   - COMPLETED

Epic 4
  - Task E4T1 --> MicroService03 -> DevTeam03  -> Estimate: 1 day   - COMPLETED

So the Epic 3, and Epic 4 are ready to be go live, but assume Epic 3 is kept hold due to further changes. Then the MicroService03 should be released only with the Epic 04 changes.

DevTeam03 can handle code branches in following ways.

1. Make fully isolated feature branches -> Can release Epic 04 easily
2. Make incremental development -> So the Epic 4 baseline also contains previously developed Epic 1, Epic 2 and Epic 3 code changes. If this is the case, delivering only the Epic 4 changes is not possible.

Assume we go with #1 and we are in Sprint 2. Now it is hard to tell which features we should merge since,

• a). More new changes are there in the 2nd sprint for MicroService03
• b). DevTeam03 only gets to know about Epic 1 and Epic 2 dependency status at the end of the sprint

Lets assume that other dependencies are completed. Then the unit tests that DevTeam03 completed in previous sprint are not valid any more due to branch merging. And also there could be code conflicts to be fixed with a thorough code review.

If we look at the estimate and draw a Gantt chart, we would easily decide the priorities of the subtasks, but it is not happening because of the task priority.

Also you can assume that there only fully isolated multiple task for MicroService03 in the first sprint, and PO/SM may decide not to release some of the changes and/or, release completed, but low priority items first.

How DevTeam03 should overcome situations like these? I personally feel that this is a way of abusing Agile. The Agile/Scrum I have learned is much much easier than this :(

Very much appreciated your comments.

UPDATE: Epic is a high level Story.

Answer 1

When I take a glimpse at what you present here my first thought is:

What about shorter sprints? Say weekly sprints. The Tasks E1T2 and E2T2 in their current state were not refined enough for weekly sprints. But if you break them up it should make planning easier. And at the end of the sprint everything would be done.

My second thought on this:

What about feature toggles?

Of course this is to be designed carefully or you end up with dozens of toggles. But the basic idea is that you are able to release a feature which is not complete by this sprint but is fully tested to the state of the current sprint. And the switch is flipped once it is ready to go live. And the switch is removed in the next release. So you decouple development from visibility. No extra branches. Of course this demands more discipline because you have to stay "backwards compatible" with the other "active" features and you have to remove the toggle afterwards to not mess up your configuration with hundreds of temporary options.

But on the other hand every work is released each sprint and nothing has to be held back from a development perspective.

The third take on this:

Why are you having independent teams working on seemingly independent features sitting (at least virtually) at one table for planning? I would have expected there were separate teams for each service with each having a separate agenda of which tasks to tackle next. The question raises whether scrum is the road you want to go down. If so, your product development has to change in such a way that the teams could work more independendly.

---

tl;dr

Your development method doesn't fit your actual work process therefore causing pain. How to solve that is up to your company.

Answer 2

There are quite a few places that I would recommend looking.

It's not exactly clear the breakdown from "Epic" into "Task". Based on my experience, it's unusual because it's missing a level of granularity. There are multiple definitions of Epic, but it seems like you are using it in the sense of a container for work. If Tasks are development tasks, you may be missing a thin slice of end-to-end, value-adding functionality that is at least demonstrable, if not deliverable. Sometimes, these are called Stories. Having this level that cuts across all of the involved services and estimating at this level may help you plan better.

The estimates also seem a bit off, with differing levels of granularity. If you're going to estimate, I am a fan of ideal time estimation. However, there's a difference between "2 weeks" and "10 days" in terms of what the estimate means. Consider what happens if you delay by 1 or 2 days. You're onto a 3rd week of development, but the 11th or 12th day. Try to be consistent with the units.

In addition to levels of granularity, a common practice is to decompose a deliverable slice of value into something that can be done in days and technical tasks as things that can be done in hours. If you use the decomposition that I mentioned above, a story that takes 3 or 4 days is on the larger side, and anything bigger should be decomposed if it's feasible. Similarly, a task that takes 12 to 16 hours is on the long side and should be broken down if it can be.

If you're going to estimate in time, I'd also make sure that you are considering ideal versus elapsed time. If you are using 2 week iterations, you don't have 10 days of productive work. There are a few different measures. Capers Jones uses something around 84-85%. In the US, with a traditional 5-day work week, 8-hour day and 40-hour week, a 2 week iteration would have 8.5 days or about 68 hours. I prefer to be a little more conservative and use 70-75%, so a highly productive 2-week iteration would be about 7.5 days or 60 hours. If you have historical data for your team and organization, using that would probably be more useful.

Once you have these tasks, map out dependencies and make sure all of the involved teams are aware of them. Then, get these teams talking on a regular basis. If a team realizes that they aren't going to get something done or if they have capacity to do more, get that information out there so the other teams can adjust.

All of this addresses planning. Making work into smaller pieces that can be aligned better to understand when things are likely to be done and helping teams know when to start or integrate work with a mainline branch so you don't have partially integrated work. However, you may still want to take extra precautions from a technical end to keep your system fully stable and integrated.

Isolating work into branches is one option. If your unit of delivery is an Epic, then you would branch per Epic. Every time the mainline branch is updated, be sure to merge it into the Epic branch to resolve conflicts and ensure all tests continue to pass. When the Epic is fully complete, it can be merged in to the mainline branch and fully integrated with the rest of the system.

You could take a slightly different take on branch per Epic by deciding where each change should go. This could even be done on a service-by-service basis. Instead of waiting for the Epic to be done, decide if integrating each story or task would keep the service in the desired state and integrate as soon as possible.

Using feature toggles or keystone interfaces can also help you integrate sooner while maintaining each service and the whole system in a deployable state. You can integrate partially-finished work while making sure it's not exposed to end users until you're ready. There is some added complexity in making sure that there are no unintended side-effects and to manage the state of the toggles, but you do get more frequent integration and less issues with resolving merge conflicts. This approach also supports trunk-based development.

You can even combine branching, feature toggles, and keystone interfaces together to control exactly when changes go in.

Answer 3

Some plans are just too ambitious for scrum/agile or any development method. You have to choose your battles in a convenient order and selectively do parallel work or risk to do go down on overhead and integration troubles.

A strong and capable product manager can make or break this. But the teams should also push back if it looks like you are set up for failure. You ultimately commit to the tasks so when it looks too complicated or dangerous you either say no or claim more time to work out any integration issues.

Answer 4

The combination of these two should make you super agile and deliver easily in small iterations by independent teams:

• Epics would in principle be broken down into small, independent and valuable stories, that are each implementable in a sprint. Prioritization at story level should help to decide what's in the sprint, independently of the epic's value.

• Microservices are meant to be loosely coupled and independently deployable. So, if E1T1 is completed, it should be releasable, even without E1T2 is not ready.

If you are in the complex situation that you describe, your microservices might not be as independent as they should. And deciding on the team's sprint backlog based on global epic ambitions instead of individual and independent stories, might simply be less agile than desired.

I'd recommend to address the root cause: revise your approach to make best use of independence and self-management of each team.

Answer 5

understand whether this is happening in the industry as a practice

That reminds me of something I saw somewhere, "I used to think Dilbert was satire, now I know that it's a documentary." Yes, this does happen other places, quite frequently, but not because we've figured out how to make it work, just because we also have confused management and bad ideas.

It sounds like in this case, though, Scrum is working as intended. Scrum (or any methodology) doesn't fix problems. What Scrum does is give visibility into problems, so that you can recognize the root causes and fix them with real fixes.

With shorter iterations and dedicated time for retrospection you can make incremental changes, test them, see if they're helping in the way that you want, adjust as necessary. It also makes certain problems much easier to document, such as in your case--it's easy to see that you have too much interdependence between teams/services and as a result work is being frequently blocked. Being able to document this is a step towards getting management changes and a step towards being able to demonstrate if changes are/aren't helping (e.g. when management says they will change and then they don't).

So this isn't failed agile quite yet, failed agile is when you've documented all of the problems and it's still impossible to change them...then it's better to go back to whatever patches and bandages work best with those problems in place. (In this case, that would probably be treating all of your services like they are one monolith and your teams are all one big team.)