1

My goal is to calculate our enterprise product team velocity in order to find process in-efficiencies and understand what could be fixed. I came across a report published by circleci that states stability, deploy time and deploy frequency to be the metrics that would help to determine engineering teams . I have a few questions when I tried to calculate the metrics for us. They are detailed as below

  1. Stability - For our application we have around 5-10 CI builds during the day. The build takes around 1 hour to execute as it runs around 25k junits and integration tests. Once this build is green further tests including a smoke test, multiple regression and performance tests are executed.

    a. Do we consider mainline to be stable only if all tests (smoke, regression, performance and client simulation tests) pass or after a CI build (that runs junits and integration tests) is green?

    b. Is stability the time spend in fixing red builds? If so, do we calculate a percentage of this time vs 24 hours? How do we include the number of builds in the calculation?

  2. Deploy time - We have a release process before which the builds are certified for production deployment. There are a few manual steps performed after all tests are green. Do we include this time in addition to the downtime taken for client upgrades?

  3. Deploy frequency - We deploy once every month for each of our customers. This frequency is part of the client agreement and not determined based on any engineering metrics. If we were to figure out the deploy frequency based on engineering teams performance, how should we be calculating it?

Thanks!

2
  • Interesting question. I'd suggest to reword the final sentence because it sounds like an opinion based question (out of scope), which is misleading since your four questions are rather objective.
    – Christophe
    Feb 22, 2019 at 8:39
  • The paper is an interesting read. However, it's more a case study of how companies use its product, rather than advise on generating your own metrics. To get proper answers to your questions, you'd need to ask circleci themselves.
    – David Arno
    Feb 22, 2019 at 9:25

3 Answers 3

2

The problem with metrics is that they are just a proxy for something, so that trying to improve some metric can have unintended consequences. This is known as Goodhart's law:

When a measure becomes a target, it ceases to be a good measure.

The classic velocity measure in a process framework like Scrum is the story points delivered per sprint. A story point indicates estimated required effort, which is related to but distinct from estimated time. A scrum team is expected to increase velocity over time because they get better at estimating the effort and better at delivering value. But if the story points were used not merely as a description of the team's work but as a metric to be maximized, the estimate would become useless: the team would be incentivized to inflate their estimates or to make features more complicated than they are.

Kanban has a more sensible productivity metric: lead time, the time from inception of an idea to its delivery. It is expected that a team will reduce lead time over time as impediments and sources of variability are identified and eliminated. This metric is very good because it is difficult to game: improvements to the metric mean that value is typically delivered earlier, and that is good. It might be tempting to game this metric by splitting up work items into smaller chunks that can be delivered more quickly, but that again is good because value gets delivered more quickly (and more consistently!). Anderson suggests using the mean lead time and a spectral analysis (histogram) of lead times, I would suggest the median lead time as an easier to interpret metric.

The Kanban book has a whole chapter that discusses metrics, for example also Throughput (similar to Scrum velocity), number of Blocked Work Items (indicates the presence of impediments), Flow efficiency (ratio of lead time to the time the issue was actively worked on), Initial quality (escaped bugs per feature), and some others.

The metrics you suggest may or may not be helpful, but note that some approaches only make sense if you are aiming for Continuous Deployment. E.g. deployment frequency is an absolutely useless metric once you reach the threshold that deployments are easy, painless, quick, and ordinary. Since you don't do Continuous Deployment but have scheduled deliveries to customers, this metric would be largely meaningless.

Your stability metrics may or may not be useful depending on how you organize development. If you use an approach like trunk-based development then you want to minimize the time during which your software is in a broken state, because that would be an impediment to the whole team. In contrast, if you use a pull request based workflow where the merge of some feature into the mainline is performed by a continuous integration server, your mainline will never be broken by construction. Instead of tracking time spent on fixing builds, the Initial Quality or Flow Efficiency metrics from the Kanban book might be more helpful.

Measuring your release process might be useful, but since deliveries are rare compared to ordinary development it would seem unlikely that changing this process could lead to a noticeable velocity improvement.

One metric that is absolutely worth considering is time to feedback from your continuous integration and automated test system. In my experience, shortening this feedback time has compounding productivity improvements across the org. Easy ways to reduce feedback time is to have these checks fail fast, to prioritize cheap but risky checks, to parallelize building and testing, and to defer high-fidelity but slow checks like expensive simulations to scheduled “nightly” builds (but also make it possible to run them on demand). It doesn't really matter whether the feedback is red or green as long as actionable feedback is quickly available so that any problems can be fixed.

1
  • Ok, so the summary is we should be looking at the agile process metrics instead of the one's mentioned in the post - at least they do not apply well for the release cycle that we have for our enterprise product. My goal with the analysis was to find out what could be done to improve the time required to release a hot fix. Do you know of case studies for enterprise applications? I am sure other enterprise product releases would be similar to us. Feb 26, 2019 at 11:43
1

It is impossible to advise you on what to measure as we're not in your shoes, but with respect to the document you mention:

Stability

The measure here is wall clock time for how long the main branch is in a broken state. This will include failed builds and possibly (depending on your build environment) failed tests.

What you deem to be a broken state will vary from project to project - only you can determine if failing test X is really a showstopper or just an annoyance.

The important thing is to measure how long it takes to get the main branch back to health and track this over time.

Deploy time

This is the wall clock time from when the build is queued to when it is in production.

My take on this is that if you have manual steps to include, these must be included. I guess the rationale here is that manual steps can themselves break and should ideally be things that are automated anyway if at all possible. Again, the important thing is to track this over time.

Deploy frequency

To me, this would seem to be the most suspect measure. By their own admission, a high deployment rate could simply indicate hot fixes for issues not found in the test cycle. It would seem prudent therefore to measure these but recognise that a deployment could happen for any number of reasons. Ultimately, it is a judgement call.

1

Nicole Forsgren, Jez Humble and Gene Kim have done the work to help us understand which software metrics are actually tied to business metrics. Through years of State of DevOps Reports and summarising their findings in the Accelerate book, the authors analysed 23,000 different profit & not-for-profit organisations of different sizes, at varying stages of digitisation.

The authors identified that, when measured against just 4 key metrics, the “highest performers are twice as likely to meet or exceed their organisational performance goals.” In other words, these indicators would lead to higher rates of profitability, market share and customer satisfaction for their respective companies. Here are those metrics:

  • Change Lead Time (or Cycle Time) - Time to implement, test, and deliver code for a feature (measured from first commit to deployment)

  • Deployment Frequency - Number of deployments in a given duration of time

  • Change Failure Rate - Percentage of failed changes over all changes (regardless of success)

  • Mean Time to Recovery - Time it takes to restore service after production failure These metrics serve as the North Star metrics for your engineering team, they are things that drive your performance whilst being fully in the domain of your engineering team

These four metrics can of course be broken down into more local metrics and risk factors, but by monitoring these as North Star metrics, you can both get a global picture and alleviate bottlenecks in your Software Development Lifecycle.

You can learn more about these metrics and how to measure them in: Beginners Guide to Software Delivery Metrics.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.