Continuous integration with testing is useful for making sure that you have "shippable" code checked in all the time.

However, it is really difficult to keep up a comprehensive suite of tests and often, it feels like the build is going to be buggy anyways.

How much tests should you have to feel confident in your CI pipeline testing? Do you use some sort of metric to decide when there is enough tests?


5 Answers 5


In General

When do you have enough automatic testing to be confident in your continuous integration pipeline?

The answer probably becomes clear if you think about what you want to be confident about. Ultimately, it maps 1-1; every test makes you confident about the one thing it tests:

  • Unit testing gives you confidence that a class (or module) does what it is tested for.
  • Integration testing gives you confidence that several units work together in the way that is tested for.
  • End-to-end testing gives you confidence that the whole application does a certain thing, the way it is described in the test.

From the way you formulated your question, you're probably thinking in a big-picture business sense right now, for example:

I want to be confident that my app can do X.

So you write an end-to-end test that tries to do X and checks if it does that correctly.

More concrete

That's all very self-referential, but that's because that's what it comes down to. There simply is not more to it.

For example, imagine you write an app to create cooking recipes. One feature is that, if you add different amounts of several different kinds of cheese, it gives you the correct temperature and time so that they all melt.

So you can write a unit test for your CheeseMeltCalculator, where you give it 100g Gouda and 200g Emmental cheese, and then you check that the temperature and time turn out right. That means you can now be confident that CheeseMeltCalculator works for 100g Gouda and 200g cheese. Now if you repeat this test with 300g Gouda instead of 200g, you can be pretty confident that it works correctly for different values. You can add tests for 0, -1 and int.MaxValue g of Gouda to be confident that the code does not trip up (or trips up correctly as intended) for weird input.

You can write an integration test to check that CheeseMeltCalculator is incorporated correctly into the whole food temperature and time calculation process. If this goes wrong, but the CheeseMeltCalculator tests above are fine, you can be confident that the bug is in other calculators or in the way the data from different calculators is combined.

And finally you can write an end-to-end test for creating a whole recipe, and one of the things you check for is the result temperature and time. If the previous 2 levels of tests are fine, but it goes wrong for this, then you can again be confident that those parts are correct and the mistake is something about how temperature calculation is integrated into the application. For example, maybe the user input is not transferred correctly.

And finally, if all of those test are fine, then you can be confident that "if you add different amounts of several different kinds of cheese, it gives you the correct temperature and time so that they all melt"

Long Story Short

The point is you can't have a test "it works correctly". You can only test "If I do X, Y happens".

However, this is exactly the stuff that should be in technical specifications for the project. A statement like "if you add different amounts of several different kinds of cheese, it gives you the correct temperature and time so that they all melt" not only gives the client clear expectations about what the finished product will do, but also can be turned into automated tests.

Additional info

User Richard added this info in an edit:

Martin Fowler has a very nice summary on his website about the most common strategies: https://martinfowler.com/articles/microservice-testing/

I don't want to remove this, but I want to say this: Compared to this answer, it is not a "summary", but rather a much more in-depth explanation, with nice graphics and everything.

My advice would be: If everything makes sense to you after reading my answer, you're done. If things still seem unclear, set a little time aside and read through the linked article.

  • This is a good conceptual view. Would you have example metrics that would be useful in providing confidence in our test coverage?
    – stonefruit
    Jul 18, 2019 at 2:28
  • @stonefruit Not really, but I think I have exactly the answer you need: Testivus On Test Coverage
    – R. Schmitz
    Jul 18, 2019 at 8:54
  • @stonefruit Regarding the number in that parable, 80%, that's a number you hear more often in this context. Mainly because of the pareto principle - the last 20% coverage is 80% of the work. In other words, it's 4 times as much work to get it from 80% to 100%, as it was to get it up to 80% in the first place. That's often overkill, but imagine you're writing control code for a satellite: If a bug pops up, you can't just fix that; getting coverage to 100% is a worthwhile investment then.
    – R. Schmitz
    Jul 18, 2019 at 9:14
  • Looks like I'm the third programmer. haha. I guess at the end of the day, it goes back to taking a risk-based approach, as you've mentioned with the satellite example.
    – stonefruit
    Jul 18, 2019 at 11:33
  • 1
    @stonefruit Maybe you are the first one, though. If you have an existing project with 0% coverage, don't start a death march to 80%. Instead, really, "just write some good tests". Maybe use the last half of Fridays for writing tests, something like that. In my experience, you will automatically come up with the tests with the best effort-reward ratio first, and every test will give you a bit more confidence.
    – R. Schmitz
    Jul 18, 2019 at 12:21

There is no metric you can calculate that will give you the confidence you are looking for. Confidence is built by doing something, and then succeeding at it or failing and learning something from it.

The only "metrics" I've found that gives me confidence in our test coverage is:

  1. Number of defects found in production
  2. Can you refactor the code base and rely on your test coverage to catch regression defects?

Automated tests are not a silver bullet. You need to keep track of how many production defects are found during each release cycle. When this number goes down, you are delivering better software. Automated tests and Continuous Integration are just tools you use to deliver better software.

The only metric you can really measure is "Are you delivering better software?"

And even then, it is subjective.

  • Compared to other answers, this answer addresses possible metrics. I've been thinking of making the suggested metrics more meaningful. Perhaps in addition to finding the number of defects found in production, give each defect a score based on risk management and place a threshold (e.g. 30 points of defects found in the past 3 months). Reaching the threshold may be indication of doing a review of the system for possible bugs, before the technical debt for buggy code increases exponentially.
    – stonefruit
    Jul 18, 2019 at 2:23

When do you have enough automatic testing to be confident in your continuous integration pipeline?

In most economic environment you will not have the budget to implement enough confidence (> 99%) but you have to manage a limited budget: It is all about the cost/benefit ratio.

  • Some automated tests are cheap to implement some are extremly costly.
  • Depending on your actual risk-management some risks must be covered by tests while other must not.

So in reality the easy/cheap/risky tests will be implemented while the expensive/unlikely tests will not.

One sub-goal of softwaredevelopment is to create architecture that easy/cheap to test (design for testability by applying Test-driven_development ) to keep automatted testing affordable.

I assume that the Pareto_principle can be applied for maintainable/testable software here, too: It says that with spending extra 20% more money you get 80% extra benefit. To reach the remaining 20% more benefit you need to spend extra 80% money.

You can apply Test Metrics like code coverage and mutation coverage to show you potential untested sourcecode.

But even with 100% coverage you cannot be shure that your code is free of bugs.

Management likes codemetrics. If "code coverage >= 80%" is enforced by management while the developpers do not support/like automated testing there are ways to write testcode with high coverage that does not prove anything giving a false feeling of security.


The trick here isn't to worry about about complete coverage but in managing the risk of your changes.

Let's say you're using your pipeline to deploy the exact same version as is already in Production - what's the risk of regression error? Zero (because there's no change).

Now, let's say I want to change a piece of text on one of the screens. I've added the test to verify that the text is now displayed correctly (let's assume for the sake of argument it's a REALLY important piece of text). What other tests do I need to verify there are no regression errors? Realistically none...

So the number of automated tests required for each release to live is not a function of the size of your application, but the size of your change. If you're making small, low risk changes then you'll need a lot less tests to mitigate the risks.

But wait a minute... doesn't this line up very nicely with the point of CI and CD?

Yep! By keeping all your changes and deltas very small you're mitigating many of the regression risks through process rather than testing. What's more, the question doesn't actually become one of automation (that's just the tool we'll use)- it's simply a matter of testing and risk appetite. Forget automation entirely, what tests would you run against a change to make sure a change hasn't introduced issues? The answer to that question doesn't change from a manual test process to a CI system - the only advantage is that many of those regression tests may previously have been developed in previous functionality and CI encourages you to make smaller, safer changes.


Your tests are to mitigate the risk of the change. A deployment with a delta of zero has no risk and therefore carries no risk. By keeping your changes small it becomes much easier to identify the tests needed to validate them - the reusability of automation is a bonus.


It is the same metric as when you're testing your product manually.

Practically, it's easy to identify these low-confidence zones: assuming that you are shipping the product, I suppose you have some post-pipeline manual steps that improves your confidence of being shippable. These are the areas you should automate to improve the confidence in the automatic process itself.

Your automation is an ongoing effort. It grows and improves as your product improves. A defect is a reason to rethink your code, along with retinking the CI. And the sunny side here is that as confidence in the product itself is achievable -- the confidence in the automation is achievable as well.

Not the answer you're looking for? Browse other questions tagged or ask your own question.