Using Kanban or Scrum across 2 companies

Question

I work for a consulting company and we're doing a software contract for a big company. Part of what we are doing for them is helping them figure out what they want, which means the requirements are pretty much in continuous update mode. We're busy speccing out version 3 while working on version 2 while fixing bugs in version 1. Right now we're using Jira in "scrum" mode only not really. We create Jira issues for bugs & improvements, move the high-priority issues into the current "sprint", and then as fires arise we move them into the current "sprint" as well (this has happened on every "sprint" in the last 18 months). We release once a week unless they hit a bug that causes enough problems that they need it fixed (and thus released) faster. So yeah, we're not really doing scrum, but it makes management happy to think this is an agile/scrum project. Management may not understand software, but they know agile is magic and makes everything better.

Our process goes like this: they create the issue in state "To Do" and assign it to me. I move it to "In Progress", make the fix or add the improvement, and move it to "Internal Test". When it's time to release, we test the issues in "Internal Test" again and if all is well we release the software and move the completed issues to "External Test" and assign it to an engineer at the client company. They test it, verify the fix, and then move it to "Done" (or assign it back to me if it's not right).

It works pretty well as an issue tracking system, with two challenges:

1. They have no incentive to test our Jira issues. Stuff sits in "External Test" for many months.
2. They want metrics to show how close we are to being complete. How do you do this while you're in the middle of creating the requirements, and thus "complete" is a moving target? Dunno.

A friend of mine said our process sounded more like Kanban, and today we tried creating a Kanban board in Jira, and now that I'm an expert on Kanban (I watched more than one YouTube video) I can see why my friend suggested it. It definitely matches our workflow better.

However, the two problems we encountered with scrum (stuff never gets tested, no metrics to show when project will be complete) seem to exist just as much with Kanban.

Are there any methodologies that work well when you have two disparate companies trying to work on the same project?

Answer 1

Scrum and Kanban aside, the fundamental problem you are facing isn't unique to your situation. It's quite common when there's a hand-off between the development organization and another organization. In your case, it's an external customer who is receiving the software, testing it, and providing feedback. I've seen similar things happen with an independent verification and validation or system integration team as well.

The general solution is to move their process outside of your process. That is, your development team should produce a high-quality product that is highly likely to meet their needs and not have blocking defects.

Whenever the client carries out their work, any feedback would go onto your backlog for future consideration. In a Scrum approach, it would be ordered with everything else in the Product Backlog, refined, and eventually brought into a Sprint. In a Kanban approach, you wouldn't have Sprints, but you'd still order the work and refine it as it gets closer to the top.

The target goal would be to be able to address the feedback at your leisure. You shouldn't have any feedback from client reviews that are "drop everything and do this now", which would allow you to take the time to order it and refine it. If you do have critical issues coming out of the client tests whenever that happens, it may be worthwhile understanding why it was a critical issue and what changes need to be made to prevent similar things from happening in the future.

The end result is that Done and all of the associated metrics happen after your development organization designs, implements, integrates, tests, and verifies that a solution is likely to be correct. Your agile approaches should help you to continually refine your understanding of the customer needs and what is necessary to have work be Done.

The incentive for the other organization to conduct their testing and carry out their feedback is that is how they know when you are done and can stop paying you to develop the system, or perhaps switch from a development-mode contract to a lighter support-mode contract. Until they accept the solution and their feedback is that no additional work is necessary (or doing more work would cost more than the value the work would deliver), then you'd continue to burn through your backlog and eventually get through the musts into the shoulds and coulds.

Answer 2

To address just your second question (as the first is covered quite good by the answer from @ThomasOwens)

They want metrics to show how close we are to being complete. How do you do this while you're in the middle of creating the requirements, and thus "complete" is a moving target?

One way to show how close you are to being complete when complete is a moving target is to use a burn-up chart.

Such a chart has the amount of work on the Y axis (in points, hours or whatever unit you use for estimates) and time on the X axis. The chart also has 2 lines. One line shows the total amount of work that means the work is complete according to the knowledge at that point in time. The other line shows the amount of work completed up to that point in time.

The gap between the lines shows how far from complete you are. The changes in the first (upper) line also show how the target of "complete" is shifting over time due to added requirements and/or revised estimates. (And if you are estimating in hours, the line could also change if the actual time to complete a work package is different from the estimate.)

Such a graph might also make it easier to digest for the customer why you are reporting 60% completion on the project last week and it dropped to 40% today after a bunch of new requirements were added.

Answer 3

...the two problems we encountered with scrum (stuff never gets tested, no metrics to show when project will be complete) seem to exist just as much with Kanban ...

This is fairly simple to address, as your true problem is not Scrum or Kanban, but rather the assumption that a Scrum or Kanban task board in Jira is enough to give you the insight (metrics) you need.

You also need (a) a high level product backlog/roadmap that can be used for estimating and reporting completion and (b) a reliable test process, within your control.

Use a product or epic roadmap to estimate and report when the project will be complete:

How close you are to completion can be monitored, estimated and reported in Jira by looking at the % of the user stories (or epics) completed in relation to the total number of user stories (or epics) in the original product requirements. Without a checklist of any kind there is no way you can estimate completion and/or total costs.

Tasks can’t be moved to “done” because things don’t get tested:

The definition of done has to be in your control. Either hire a test manager to make sure it gets done and is in with your control, or remove the “externally tested” step from your board. Board metrics are only useful if they measure your performance in a process you control, so that you can predict future performance for tasks in a process you control.

You sort of answered your own question when you said:

... we're not really doing scrum ...

An underlying assumption behind both the Scrum board and a Kanban board is that the tasks can and shall be completed by the team/resources using the board. A velocity is only a useful metric if can be used to estimate the future performance of those same team resources, against a known set of goals.

The Kanban board assumption is that the tasks are of a similar size, without much uncertainty, and helps you measure throughput. Again, knowing your throughput is only helpful in estimating completion if it gives you your team’s throughput per resource. An estimation of completion or cost for a set number of similar tasks must assume that “done” is within your control, and that throughput will be fairly consistent.