I'm researching DevOps metrics for school. One metric is called "automated test pass percentage." I found one article that says this metric is useful because it is good to know how often code changes are causing your tests to break. Does this metric tell you how many tests pass in your pipeline whenever code is committed?
This metric, to determine what percentage of tests pass, doesn't make much sense to me.
There are two reasons why a test would fail:
- The test found an error. Some change in code caused the behavior under test to change and the test is alerting you to a case that needs to be considered. The change needs to be reworked to ensure that the system has the appropriate external behavior.
- The test needs to be updated. The change caused the test to become invalid. The underlying change was valid and correct, but the assumptions or assertions made in the test are no longer correct and need to be revised.
Ideally, you catch all of 2 when you make the change. When you open up a pull request or merge the changes into a branch, all of the appropriate test changes are also done. However, depending on the size and complexity of the system and the implementation of tests, it may not be feasible to run 100% of tests on all changes all the time. You may be forced to choose the tests to run based on what parts of the system are changed versus what the tests cover.
I'm just having a very difficult time understanding how you could get value of out this metric. The only thing that I can think of is that it's low-hanging fruit and most CI tools can track this relatively simply, but I'm having difficulty coming up with a good way to use that data.
Automated tests in general will be a unit tests and acceptance tests. Test percentage is really important factor. Whatever the code that developer had written must be covered with unit tests.
That's said it has to be 💯%
Some might think it as unrealistic. Any code that we write is to execute in our application. If it needs to be executed currently, it has to be tested. Of course we cannot rely on manual testing all time. It is really necessary to have our tests coverage 100% always.
When it comes to pipeline, there are different tools available to achieve your expectations.
Once you commit your code, pipeline gets triggered automatically which pulls the code from the repository, builds it, runs unit tests, deploys it in respective servers, runs acceptance tests and provides you the basic metrics of how many tests are executed, no of failure tests etc.
When you integrate these pipeline with the some static analysis tools like Sonarcloud / sonarqube, it gives you a excellent metrics / bashboards on how many tests has been executed, how many failures, code coverages, code smells, potential bugs etc.
Some of the popular pipeline tools are, Jenkins, Teamcity, AWS Code pipeline etc.