How to deal with (or account for) time lost in waiting for other things in an Agile environment?

Question

We use Scrum. We have a month long sprint. In the beginning of the sprint we estimate tasks from the backlog and then depending on what capacity we have, we add tasks to the current sprint.

However, the nature of our work is such that (we develop stock market connectivity solutions) we often have to wait for long periods of time (sometimes half a day, and almost 2-3 days in extreme cases). This happens because the stock exchange might have screwed up something on their side, they might not have communicated something because of which something broke on our side and we are trying to clarify it with exchange, exchange might be late on delivering some functionality which ends up creating delays on our side.

The problem is that, let's say there was a task X for which I estimated Y hours. Now I started working on X, but after some time, I am required to talk with the exchange people and they are late to response. Now the time taken to complete X is not Y, it is Y + Y'.

How (or where) do I account for Y'? Technically, it still took me Y amount of time to complete the task. The time Y' was basically spent idle. (I generally use Y' to do several things like writing some scripts that improve productivity .. some trials to see if I can improve latency of the product, etc). My manager knows this and he is more than happy with the situation. However, from the Scrum/Agile point of view, I face trouble. How do I fix this situation?

Answer 1

Also from the Agile/Scrum viewpoint, you are not facing real trouble.

First of all, in the Agile world it is widely accepted that estimates in hours have to be taken with a large grain of salt and relative estimates in points with an only marginally smaller grain. Estimates are by definition going to be inaccurate.

What is valued most in an agile world is transparency. In your situation, this can be given in several ways, depending on the circumstances. For example, if you had a large delay (more that half the average task length), you can mention that in the daily standup and if you found some other useful work to do while waiting, mention that as well.
Also, if you are waiting for an external party for more than half a day, you should make that visible on the task board.

If the delays due to communication/interaction with external parties are frequent, you should try to take that into account when planning your stories/tasks by reserving time for the expected average delays. If the size of this "delay buffer" roughly matches the total delay time that your team encounters in a sprint, the predictability of the amount of work that you can handle will increase.

Answer 2

The way I have seen this done successfully is to separate the actual integration piece into a separate story:

1) story 1: app does it's work, sends data to the interface, automated testing framework tests that the app is sending data correctly. (and it tests responses / returned data to make sure the app is working properly with the response / data returned).

2) story 2: with story 1 complete, this is an integration testing story to make sure the interface is working properly. What is properly you might ask? Well, there must be a contract of some interface (probably a json object definition or something with a description of how it works), if there's not then that's the real problem. Put all of your work that has external dependencies in a separate story. Estimate what you think it will take to test and you'll start trending out how well you / your team are managing the external dependencies. Btw, i did work for stock exchanges too and this will work with them.

You might say this is waterfall, but it is actually a good agile process - you've separated the things you have no control over from the things you do. After doing this you've proven that your app works end to end with the integration based on how you understand the interface - you've met your product owners objectives. If you find out it does not work once you've integrated then your story did not have correct info and the ramifications of the changes need to be worked through from a user perspective. This might result in new functionality being added into your app (or not). If this occurs over and over again with the same external team, then there needs to be some work on the process in uncovering the "real" integration requirements with that team.

Also, in this area - i see the external integration team as a "user" (persona) that you need to understand very well for an ongoing successful integration. It may not be a product owner that uncovers how to do this but a developer, regardless, that's where I would spend time and focus.

Answer 3

I almost laughed my head off when a project manager once told me to pad out my hours to hit "40hrs for the week".

Lets say task Y was estimated to take 8 hours and you spend 6 hours over two days. What should you report?

If you report 8, then the next time a similar task needs to be estimated, they won't know the actual time. It's even worse if you report 16-hours. Do this too often and it will look like you are 'cheating' the company.

So you must report real time against the actual task. Your PM needs to be aware of this "non-billable" time, especially if it's because of a blocking item (the communication). Maybe the delay is unacceptable and needs some pushing. In some cases the turnaround for responses is set contractually. Your company cannot offer a fair price for their service if half the company is just sitting around not doing billable work.

Assuming you are on a salary, then you should offer to fill in on some other task that helps the company.

If this happens rarely, maybe there is some manual monthly report you could automate for yourself or somebody else.

If it happens 'now and then', study up on some new framework -- offer to teach the rest of the team next week when they are all sitting around.

If this happens a lot, you could offer to study for a certificate of some type. That will help the company and help you.