is there any elegant way to analyze an engineer's process?

Question

Plenty of sentiment exists that measuring commits is inappropriate.

Has any study been done that tries to draw in more sources than commits - such as:

• browsing patterns
• IDE work (pre-commit)
• idle time
• multitasking

I can't think of an easy way to do these measures, but I wonder if any study has been done.

---

On a personal note, I do believe that reflection on one's own 'metrics' could be valuable regardless of (or in the absence of) using these for performance eval. I.E. an un-biased way to reflect on your habits. But this is a discussion matter beyond Q&A.

Answer 1

Not sure if you would consider it elegant, but Watts Humphrey wrote an entire book called Personal Software Process that was all about measuring your own productivity. He outlined metrics for inputs like time at your desk vs. interruptions, time spent working on various kinds of software life cycle activities, defects per amount of code. There is a technical report that gives the short version at:

http://www.sei.cmu.edu/library/abstracts/reports/00tr022.cfm

If you want to look at something like the quality of a developers code, Chidamber & Kemerer proposed several metrics for object oriented code.

Metrics for object oriented code

• depth of inheritance tree,
• weighted number of methods,
• number of member functions,
• number of children, and
• coupling between objects.

Using a base of code, they tried to correlated these metrics to the defect density and maintenance effort using covariant analysis. Later studies did similar analysis on hundreds of Source Forge Java projects to determine their characteristics relative to CK Metrics and some additional metrics proposed later.

Metrics arising in context of Code Reviews

Defects can be categorized by many criteria:

• severity: (major, minor, cosmetic, investigate/unknown),
• type (logic, typo, spelling, coding standard violation, etc.),
• origin/phase containment (requirements, design, code, etc.).

There are also preparation and inspection rates (time per reviewer, time per line of code) and defect densities (per KLOC (thousand lines of code), per minute of inspector/reviewer time).

Plotting these values against control charts can show us whether we are within bounds for the process (for example, a team that inspects 200 lines of code that finds no defects in a group that averages twenty-five defects per KLOC might be malfunctioning).

Other metrics

Other metrics that could help include those for

• earned value to help calibrate estimation
• velocity for those using Scrum or Kanban.
• The work of Pareto, Juran, Demming, Ishikawa, and others from quality and industrial engineering is somewhat portable to software development with a little bit of imagination.

Limitations of Analysis

There are tremendous limits on the value of metrics. Bugs fixed per developer can mean almost anything, and when you start to punish or reward on that measurement, I bet bugs will get more numerous and granular, and the mix of difficult to easy bugs fixed will also change as the team cherry picks in the to race to have the most.

There is also a certain distraction and potentially a loss of focus and enjoyment that can come with intrusive measurement. You can't get much more elegant (and emotionally burdened) than a lake poet like Wordsworth who said,

      Sweet is the lore which Nature brings;
      Our meddling intellect
      Mis-shapes the beauteous forms of things:--
      We murder to dissect.

While software isn't exactly Nature, give me some latitude because I thought I would never get to use anything from high school English Literature class.

Agile might be considered a reaction to centralized, big process. Sometimes that mode of work could require so much analytical effort that the ability to reach flow while creating software all but disappeared.

Answer 2

I want to add an alternative answer that points away from standard software engineering practice toward another field with the aim of stealing from basic tools that we can adapt as needed. Quality assurance people are concerned with production, yield, and defect detection and prevention, just like software developers.

http://en.wikipedia.org/wiki/Seven_Basic_Tools_of_Quality

I like the control chart.

http://en.wikipedia.org/wiki/Control_chart

Do an activity, plot a metric, do another, plot its metric, and so on. For example, plot commits per day. The chart will scatter data that ranges from some minimum to some maximum. Perhaps later you can characterize the results to determine that when the value is low, something is impeding progress and when it is too high, work is begin done in a fast but sloppy way. How you encourage workers to speed up or slow down is left to you.

Personal metrics might be something you can correlate for yourself starting with a question like, "I feel most productive when I..."

• Write a complete use case before start to code.
• Write my unit tests before my code.
• Check with stakeholders frequently to insure requirements don't change and create massive rework on work done toward an obsolete plan.
• Change as much code as possible.
• Delegate well defined changes to team members who are most expert with the parts I asked them to change.
  • Give my team a complete overview, but with priorities we can finish in the current sprint.
  • Start my refactoring pass with a hierarchical list of directories, files, classes, methods, equations, variables, documentation, etc. that I will change.
  • Research a high level problem to find prior art solutions, estimating the scope and key improvements needed to make a better solution.

The old saw what we measure is what gets done might lead you to attack the problem based what you determine to be the limiting factor

or multiple factors in order of priority based on a Pareto diagram.

http://en.wikipedia.org/wiki/Pareto_chart