I'm researching DevOps metrics for school. One metric is called "automated test pass percentage." I found one article that says this metric is useful because it is good to know how often code changes are causing your tests to break. Does this metric tell you how many tests pass in your pipeline whenever code is committed?

  • 1
    Based on the description "it is good to know how often code changes are causing your tests to break" I'd guess that isn't the coverage measure described below, but is the percentage of builds that fail. – jonrsharpe Oct 5 '19 at 7:00
  • Do you have a link for the article that says this metric is useful? I'm having a hard time understanding that. I think it's not unexpected to have failing tests - it may be difficult or slow to run all unit, integration, and acceptance tests on every branch on every commit (or pull request) - you run the tests that increase your confidence sufficiently to proceed. A failing test could mean that a defect was introduced or that a test needs to be updated for the changed behavior. – Thomas Owens Oct 5 '19 at 10:35
  • Thomas Owens, here is the link to the article. It's the sixth metric discussed. stackify.com/15-metrics-for-devops-success/… – Marcus Kim Oct 5 '19 at 22:40
  • @MarcusKim counting the number of times the build doesn’t pass is a pretty useless metric. Worse than useless maybe, measuring this could be actively harmful to a team. The only time I would worry about this is if the tests never fail, because that means they don’t really test anything. Teams who are afraid to see a build go red don’t deliver well. They don’t understand that it’s good when a build goes red. It means we didn’t ship a broken program. – RubberDuck Oct 6 '19 at 9:15

This metric, to determine what percentage of tests pass, doesn't make much sense to me.

There are two reasons why a test would fail:

  1. The test found an error. Some change in code caused the behavior under test to change and the test is alerting you to a case that needs to be considered. The change needs to be reworked to ensure that the system has the appropriate external behavior.
  2. The test needs to be updated. The change caused the test to become invalid. The underlying change was valid and correct, but the assumptions or assertions made in the test are no longer correct and need to be revised.

Ideally, you catch all of 2 when you make the change. When you open up a pull request or merge the changes into a branch, all of the appropriate test changes are also done. However, depending on the size and complexity of the system and the implementation of tests, it may not be feasible to run 100% of tests on all changes all the time. You may be forced to choose the tests to run based on what parts of the system are changed versus what the tests cover.

I'm just having a very difficult time understanding how you could get value of out this metric. The only thing that I can think of is that it's low-hanging fruit and most CI tools can track this relatively simply, but I'm having difficulty coming up with a good way to use that data.

| improve this answer | |

Automated tests in general will be a unit tests and acceptance tests. Test percentage is really important factor. Whatever the code that developer had written must be covered with unit tests.

That's said it has to be 💯%

Some might think it as unrealistic. Any code that we write is to execute in our application. If it needs to be executed currently, it has to be tested. Of course we cannot rely on manual testing all time. It is really necessary to have our tests coverage 100% always.

When it comes to pipeline, there are different tools available to achieve your expectations.

Once you commit your code, pipeline gets triggered automatically which pulls the code from the repository, builds it, runs unit tests, deploys it in respective servers, runs acceptance tests and provides you the basic metrics of how many tests are executed, no of failure tests etc.

When you integrate these pipeline with the some static analysis tools like Sonarcloud / sonarqube, it gives you a excellent metrics / bashboards on how many tests has been executed, how many failures, code coverages, code smells, potential bugs etc.

Some of the popular pipeline tools are, Jenkins, Teamcity, AWS Code pipeline etc.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.