5

A few sprints ago I was assigned a task that was primarily research. I had to figure out how to get our product to interoperate with a very complex black box that we did not develop.

I couldn't think of a way to estimate this work. Even if I got the ball rolling and knew the immediate problem I faced, I could not get a sense of how many other problems I'd have to solve after that. I could never tell if I was almost done or far from it. How am I supposed to estimate a backlog item like this?


I want to elaborate the nature of this assignment. I knew what calls I had to make to interoperate with the black box. That was the easy part. But the API took a very, very complex object as a parameter. Calling the API would throw an error and it was not easy to figure out what that error was trying to tell me. The black box wouldn't tell me all the problems wrong with my request, it would just tell me the first problem it found. This made it very difficult to know how much work I had left.

  • Of course you can come up with an estimate, just don't expect it to be a very good one. This has to fit in with your primary goal of estimates. If for example, you use them to help plan future work, this type of unique task isn't going to be very helpful anyway. For client billing, this does present a problem, but so does this project. – JeffO Jan 23 '14 at 0:13
  • 1
    It certainly isn't pure SCRUM, but I've started to associate not only an estimate but a risk/estimate inaccuracy factor 0-3 with backlog items. Almost everything is 0, but this helps me communicate clearly to project managers etc. that our team is unsure of an estimate. The nice thing about this is that it can, if so desired, feed into more complicated confidence models about time estimates - but we've never gotten to that yet. I think the underlying issue is that there are truly at least two dimensions here: time and confidence. – J Trana Jan 23 '14 at 2:31
15

if you can't estimate - and in this scenario it sounds like there really is no way to know in advance how long any particular part of the process will take - then the next-best option is to time-box the effort: how much time you're willing to spend on it, whether you get anywhere or not

once you get into it, you may have a better idea of how to estimate the remaining effort

  • 2
    This is what we do for research/investigation. We limit the research to, say, a couple of days, then reduce the number of points we pull into a sprint in exactly the same way as if someone had taken those as vacation days. – Izkata Jan 23 '14 at 3:52
5

Incredibly simple: It's always 3 points.

(or pick a different constant). This is the approach we took at our last job and I think it worked out relatively well.

The idea is that you have a backlog full of actual stories and one of them happens to be an integration story with a device you know nothing about. If someone asked me to size that story(actual integration) without any more knowledge, I would give a very conservative estimate. If I did similar integrations in the past, I might pick a story points on high end. If I never did similar work and don't even know where to begin, I would just pick a million points.

So then obviously product owner comes back and says, we can't have a million points, so let's create another story about research. The goal of the research story IS NOT to code the integration. It IS NOT about solving every possible problem you might face. The primary and only goal of the research story is to give better point estimate for the actual work story. Don't get lost trying to figure out how to use every single API call and come up with actual integration design/code. Instead, approach it from a high level to understand the complexity that's ahead of you.

At the end of the research story, your #1 deliverable (along with possible research notes, prototypes... etc) is a point estimate for the integration and possibly break down of integration story into several smaller parts. Keep in mind that each of those still has an unknown component given enough research, you should be able to control for that unknown part (i.e. you read about usage of some API and it looks like its going to be 5 points, well, pick 8 just to be safe).

What we found is that the developer assigned the research task always runs out of steam at almost a constant number of hours (obviously there is some fluctuation, but it averages out at the end). After that you could spend twice as many hours, but your integration story estimate won't get any better.

3

That's why they call them estimates.

An accurate estimate for purely research-related work is improbable. That's the nature of research. But I think it's worthwhile to point out that sometimes you think you can't estimate it, when in actual fact, you can. Scientists know this; most research work is repetitive and mundane... it's the theorizing and changes in direction that cause variations in the required time to occur.

So the way you come up with an estimate for research-related work is the same way you would come up with an estimate for ordinary development: break the task into smaller pieces, and evaluate each piece separately. Research work is no different in that regard from ordinary, non-research development; you quantify the work, and then give it your best guess.

For example, you might not be able to estimate the task "Determine how this black box works, and how to interface our software product with it." But you may be able to come up with independent estimates for:

  1. Evaluate and understand the black box's API
  2. Evaluate the behaviors of the black box
  3. Determine what commands will execute the behaviors we want
  4. Analyze, identify and mitigate possible safety concerns
  5. Map commands from our software to commands in the black box, and evaluate the returned data.

And so forth.

Some estimates may depend on the completion of previous steps. For example, estimating step #3 may not be possible until step #2 is completed.

  • In my situation it wasn't really possible to break things down this way. I knew which API to call. The majority of the time was spent deciphering and fixing the errors it was giving me on that call. But I can see how for general research problems this is a good approach. – Daniel Kaplan Jan 22 '14 at 22:38
  • In XP (and often used by Scrum practitioners) they call this a Spike and they try to timebox it. – Dave Hillier Jan 22 '14 at 22:42
  • -1: What you propose is simply breaking down a large task into smaller ones. The OP has no idea what the task is, let alone how big it is and how to break it down. If I ask you to estimate how long it will take to get from towns Acme to Emca, whats the approach? – mattnz Jan 23 '14 at 1:39
  • @mattnz: Then I refer you to the first sentence in my answer. I did suggest that an accurate estimate is improbable. That's the nature of research. But I think it's worthwhile to point out that sometimes you think you can't estimate it, when in actual fact, you can. – Robert Harvey Jan 23 '14 at 1:39
  • Expand that first line answer (as that is really the correct answer), and the edit will allow me to reverse the down vote.... – mattnz Jan 23 '14 at 1:49
2

Two ways I'm using:

The deliberate discovery approach: start with two stories (in reverse order):

  • how much time you'll need to figure the system out
  • how much time you'll need to estimate how much time you need to figure the system out - keep that one very short on purpose

The key is that any of those is limited by the value of the feature. It doesn't make sense to spend a year figuring out a system in order to deliver feature X, if in half that time you could deliver feature Y and Y has more business value. Your PO should be able to give you a number: 'if you need more than two months to figure it out, it's not worth it'. Refine the estimate for the first story when you complete the second one.

The discover as you go approach:

how to get our product to interoperate with a very complex black box that we did not develop.

This is way to big a story. What is the first thing/feature/aspect you're going to work on (which one is the most valuable?). It might be as simple as 'log in the black box'. This is a simple story, the integration points shouldn't be that bad, and is reasonably easy to estimate with minimal research. Estimate that. Put any other story at it's business value boundary and refine as you learn more.

Either approach is problematic regarding long term planning. When you get started, you just can't tell when you'll have all features done. That's ok - you'll just start by having reliable predictions for tomorrow, then next week, then next month, then 6 months as you build expertise.

2

Estimates are not single numbers. An estimate is a range that has a high end, a low end and a confidence. With the initial question of "how long will it take you" an estimate is not "it will take me 14 hours" but rather "it will take me between 10 and 18 hours".

As you refine the process, you can get a tighter and tighter range on the estimate


(source: construx.com)

Note that this shows the variability of the estimate, not that "it converges at the average". An estimate may start out at "it will take me 10 to 18 hours" and then be "it will take me 14 to 18 hours" and then "it will take me 16 to 18 hours" as you get further along in the task.

One of the most often cited books in estimation is Software Estimation: Demystifying the Black Art by Steve McConnell.

Start out with a very wide estimate that has the best case and the worst case. Then shortly into the research, do another estimate, and then another and another. Don't make a commitment as to when it will be done with one number and then try to make it.

There are a number of different techniques for generating an estimate. Use historical data from past estimates. Try to get an idea of how big it is, the complexity and work from that (the list goes on, there's an entire book written about this).

  • 1
    I think this is good advice in general. But like I said, "I could never tell if I was almost done or far from it.". – Daniel Kaplan Jan 22 '14 at 22:40
  • A tangental read on the subject - Steve trying to build a tree house: construx.com/10x_Software_Development/… (mentioned in codinghorror.com/blog/2007/09/… ). But its still a good read and still important to figure out how to estimate and if there are techniques that may be more appropriate to one approach that aren't appropriate to another. – user40980 Jan 22 '14 at 23:05
0

While many research tasks can be open-ended, this one has a clear(ish) requirement: "how to get our product to inter-operate with a ... black box".

We are good at estimating jobs that are similar / related to jobs we have done before. Since you haven't done it before, you can only break the job into chunks and give each chunk a time-box. Such a job can go on forever, at the end of each time-box progress and tasks remaining should be reevaluated. In extreme cases this is when a go / no-go decision should be made.

When I work on such jobs, I define a 5 step process

  • (1) Initial planning. Make an initial guess at (2) what needs to be done to get started, and define (3) a bunch of iterations / stages, each followed by (4) a review and (5) re-plan step. For example:

  • Getting started tasks. In this case it might be "get the black box specs", "study the specs", "obtain a black box", "obtain sample software", "establish support link".

  • Iteration 1. Maybe "Test basic operations. Setup basic debugging environment. Review. Re-plan."

  • Iteration 2 ... and so on, with at least 3 but no more than 5 or maybe 7 iterations before a big progress review.

How long will each iteration take? Time box them - such jobs will go on forever if you don't. Only you can say if the boxes should be an hour, a day, a week, ... .

Also, be flexible: as you learn, the plan may have to change. Don't punish yourself. This kind of job is like debugging - you have to teach yourself. But also like debugging, it can help to get input from others. And like debugging, you cannot do it while multi-tasking.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.