How to measure time lost due to environment problems?

Question

I'm trying to determine how we can measure how much time the developers are loosing to poorly managed environment and inefficient ways-of-working.

The goals are:

1. Show managment how big the problem really is, in terms they can relate to. I.e $$$
2. Get a good baseline so that we can measure the progress.

Short background:

We manage the programming environment for 500 developers. This includes IDE (mainly Eclipse and Visual Studio), version control systems (ClearCase), what programs they have available. Basically, everything related to the computer that involves programming.

Examples:

Not being able to work with tools they are used to due to strict security requirements. Using version control system in a bad way causing problems when working in parallell. Setting up project environment is done manually and different each time. Using homecooked frameworks, rather than what are industry standard. Building takes longer than needed due to homemade practices. Deploying takes longer than needed due to homemade practices.

This does not include the actual coding pratices, project management or meetings. Sure, we can influence those things too, but for this questions those things are out of scope.

The question:

So, how can we determine how much time are lost due to the project environment?

The only thing I can think of to get some good data are polls and questionnaires.

Have anyone done something like this? What were the experiences and results?

Are there other literature or reasearch where I can get ideas from or compare results with?

Other questions exists regarding programmers inefficiency (like this), but those relate to coding practices and bugs.

Answer 1

What it appears that you're doing is attempting to measure productivity. Measuring productivity is notoriously difficult. Here's how I go about it...

I've found that you don't want to tackle the situation by measuring losses, you want to measure the gains of potential solutions. From a purely philosophical point of view, the former implies that you're doing something wrong, which may not be true.

Things change, and periodically, someone (like yourself, it seems) needs to take a look at the landscape and see if those changes can provide a better environment. With that said, I look at:

Things That Don't Exist

• Tools (e.g. computers, version control)
• Practices (e.g. TDD)
• Processes (e.g. code reviews, agile)

This is a common problem and I've successfully argued in favour of adopting certain things based on:

Product Documentation and Sales Information

This is always the first place to start. If the product costs money, the sales documentation is almost always targeted or worded for management types. It's easy to find, easy to access, and a human that works for the vendor is almost always available to provide other resources like the following...

Case Studies

A good example I have is the book "Best Kept Secrets of Peer Code Review" was written by Jason Cohen (founder of Smart Bear IIRC) contains case studies in order to make the case for code reviews. Sure, the book was basically written by the vendor, but the case studies were legitimate.

Proofs of Concept

If it's a potential implementation (thinking CI, build system, TCM, or any other process or automation facilitating tool) then the best way to go about it is to get a trial and set it up. After performing some initial investigation (ie, using it on fake projects) it might be a good idea to try it out in a small project, or possibly in tandem with the existing solution.

For processes, it's similar. This is basically the way that a lot of developers introduce unit testing. They maintain their own tests for a while, and when the critical "BAM! Regression Caught!" scenario hits, they perform the reveal and show how it saved the project. A little dramatic, yes, but it makes sense. People need to see the benefit with their own eyes.

Things That Exist but are Broken/Inefficient

This will almost always be a comparative analysis. If you approach it that way, then you should be able to get numbers to work with.

Sales info from the vendor doesn't always work well here, unless it's a product that directly and obviously competes with the currently implemented solution - in that case, you can usually get the vendor to help you out here. (I'm thinking SourceGear Vault vs. SourceSafe as an example)

I've found that case studies rarely exist that compare tools/practices but do exist for processes (think Agile vs. Waterfall studies).

In my experience, proofs of concept are often the most successful ways of converting management/developers from one tool or practice to another. They have to see it working, and more importantly, see it working better than what already exists. Then you hit them with the numbers. "This system, which is only a POC, is still x times faster than our previous one. That translates to y reduced man-hours of work and z reduced overhead" Remember that POCs can sometimes be done on small projects to test out a process, usually after sufficient evidence of potential gains has been given.

Things That We Have No Idea About

Here's where your polls and questionnaires come into play... but these will lead you to the categories outlined above.

Answer 2

I assume you mean environment downtimes and lost productivity when development and test environments go down or need developer time to debug environment issues.

Make a simple tracking tool that is so easy to use that developers don't have to think about it. Whenever an environment issue is being debugged by a developer, or they simply cannot work because of it, I had created a stopwatch like tool with something simple like AutoHotKeys (a tool that allows you to quickly and easily map key combinations to a script) that started a timer. When the downtime was complete, hit the key combo again and AutoHotKeys send a simple GET request to a PHP page that I had set up on the webserver. It just tallied up totals in milliseconds and stored them in a file.