When do you have enough automatic testing to be confident in your continuous integration pipeline?
The answer probably becomes clear if you think about what you want to be confident about. Ultimately, it maps 1-1; every test makes you confident about the one thing it tests:
- Unit testing gives you confidence that a class (or module) does what it is tested for.
- Integration testing gives you confidence that several units work together in the way that is tested for.
- End-to-end testing gives you confidence that the whole application does a certain thing, the way it is described in the test.
From the way you formulated your question, you're probably thinking in a big-picture business sense right now, for example:
I want to be confident that my app can do X.
So you write an end-to-end test that tries to do X and checks if it does that correctly.
That's all very self-referential, but that's because that's what it comes down to. There simply is not more to it.
For example, imagine you write an app to create cooking recipes. One feature is that, if you add different amounts of several different kinds of cheese, it gives you the correct temperature and time so that they all melt.
So you can write a unit test for your
CheeseMeltCalculator, where you give it 100g Gouda and 200g Emmental cheese, and then you check that the temperature and time turn out right. That means you can now be confident that
CheeseMeltCalculator works for 100g Gouda and 200g cheese. Now if you repeat this test with 300g Gouda instead of 200g, you can be pretty confident that it works correctly for different values. You can add tests for
int.MaxValue g of Gouda to be confident that the code does not trip up (or trips up correctly as intended) for weird input.
You can write an integration test to check that
CheeseMeltCalculator is incorporated correctly into the whole food temperature and time calculation process. If this goes wrong, but the
CheeseMeltCalculator tests above are fine, you can be confident that the bug is in other calculators or in the way the data from different calculators is combined.
And finally you can write an end-to-end test for creating a whole recipe, and one of the things you check for is the result temperature and time. If the previous 2 levels of tests are fine, but it goes wrong for this, then you can again be confident that those parts are correct and the mistake is something about how temperature calculation is integrated into the application. For example, maybe the user input is not transferred correctly.
And finally, if all of those test are fine, then you can be confident that "if you add different amounts of several different kinds of cheese, it gives you the correct temperature and time so that they all melt"
Long Story Short
The point is you can't have a test "it works correctly". You can only test "If I do X, Y happens".
However, this is exactly the stuff that should be in technical specifications for the project. A statement like "if you add different amounts of several different kinds of cheese, it gives you the correct temperature and time so that they all melt" not only gives the client clear expectations about what the finished product will do, but also can be turned into automated tests.
User Richard added this info in an edit:
Martin Fowler has a very nice summary on his website about the most common strategies: https://martinfowler.com/articles/microservice-testing/
I don't want to remove this, but I want to say this: Compared to this answer, it is not a "summary", but rather a much more in-depth explanation, with nice graphics and everything.
My advice would be: If everything makes sense to you after reading my answer, you're done. If things still seem unclear, set a little time aside and read through the linked article.