16
In General
When do you have enough automatic testing to be confident in your continuous integration pipeline?
The answer probably becomes clear if you think about what you want to be confident about. Ultimately, it maps 1-1; every test makes you confident about the one thing it tests:
Unit testing gives you confidence that a class (or module) does what ...
7
Ask yourself what unit tests are good for. The main purpose is that after changing your code, you run all the unit tests, and if they all pass, you have a bit more confidence that your code is fine. For this purpose, what matters is the number if independent assertions that you passed, so for this purpose a single unit test with 100 assertions is no problem. ...
4
Each unit test should assert a single requirement.
Now, you may not always have formal requirements for every method. Methods are written in pursuit of fulfilling such formal requirements. But given that a method should do only one thing, should have a single purpose, you should already have a fair idea of what to expect from the method in terms of ...
4
There is no metric you can calculate that will give you the confidence you are looking for. Confidence is built by doing something, and then succeeding at it or failing and learning something from it.
The only "metrics" I've found that gives me confidence in our test coverage is:
Number of defects found in production
Can you refactor the code base and rely ...
3
Testing with stubs/mocks/fakes is good, but it can never fully replace system and integration tests where the real thing gets used.
If you don't have enough control over the external service to reliably use it in an automated integration test, then you should perform a manual integration test before you merge your work to the 'master' branch to prevent just ...
3
Is this use of terminology something unique to this person?
I have no idea if this specific use of terminology is unique or not, but it is not uncommon to find teams or persons in the software industry using terms in a quite unusual, non-standard, or imprecise way.
For example, due to the popularity of tools like JUnit or NUnit, I have very often heard ...
2
A unit test is an (almost always) automated test which verifies the behaviour of a small, isolated unit of code.
This unit of code is usually a single method or function. Any more than that and you start getting lots of scenarios. There are normally multiple tests verifying a single piece of code with each testing a specific scenario.
Generally a good unit ...
2
They are completely different.
Unit testing is concerned with verifying that small chunks (functions/methods) of code work in isolation. They should test cases of common usage, edge cases and any case in which an error can occur. For example, if you have a function that adds two numbers, then your unit test must test whether or not the function calculates ...
2
When you are writing mocks for the upstream services you depend on, you are matching the service's behavior for a limited number of cases.
If you get a 4xx error from the service in production, it means your mocks are not accurate to the service's actual behavior or the upstream service introduced a breaking change.
The best way around this is to checkout ...
2
Not having separate QA is a very bad thing. At the most basic level, the software developer wants the software to work to demonstrate they did a good job, and the QA person wants the software to not work to demonstrate they did a good job. You can't do both at the same time.
2
Is trackUserPerformance within the data access layer?
If no, then you should consider adding the layer. A common—although not the only one—approach is to separate applications into presentation layer, which handles all the UI logic, business layer, which contains the actual logic of the application, and the data access layer which provides persistence.
If ...
2
When do you have enough automatic testing to be confident
in your continuous integration pipeline?
In most economic environment you will not have the budget to implement enough confidence (> 99%) but you have to manage a limited budget: It is all about the cost/benefit ratio.
Some automated tests are cheap to implement some are extremly costly.
...
1
If you are concerned about an integration test that interfaces with an unreliable external system, then one thing you can do is have a separate testing phase on your automated tests for the external test. This should not fail the whole build, if this phases fails (it may indicate a bug or it may indicate the downstream system has gone down), but indicates ...
1
Exercise the public interface of all classes and all utility functions. Doesn’t matter how “small” the feature is, you can still get regressions.
1
The trick here isn't to worry about about complete coverage but in managing the risk of your changes.
Let's say you're using your pipeline to deploy the exact same version as is already in Production - what's the risk of regression error? Zero (because there's no change).
Now, let's say I want to change a piece of text on one of the screens. I've added the ...
1
You are looking at a combinatorial problem:
You have 3 columns, so you need all combinations of 3 columns: [[],[1],[2],[1,2],[3],[1,3],[2,3],[1,2,3]] where [ and ] enclose a list of included columns. And yes, [] is a valid combination.
You have 2 filters available for each column, plus an unfiltered state. You also have ensure that you have data that ...
Only top voted, non community-wiki answers of a minimum length are eligible
