I can't point to a good online resource (the English Wikipedia articles on these topics tend to be improvable), but I can summarize a lecture I heard which also covered basic testing theory.
Testing modes
There are different classes of tests, like unit tests or integration tests. An unit test asserts that a coherent piece of code (function, class, module) taken on its own works as expected, whereas an integration test asserts that multiple such pieces correctly work together.
A test case is a known environment in which a piece of code is executed, e.g. by using specific test input, or by mocking other classes. The behavior of the code is then compared to expected behavior, e.g. a specific return value.
A test can only prove the presence of a bug, never the absence of all bugs. Tests put an upper bound on program correctness.
Code Coverage
To define code coverage metrics, the source code can be translated to a control flow graph where each node contains a linear segment of the code. Control flows between these nodes only at the end of each block, and is always conditional (if condition, then goto node A, else goto node B). The graph has one start node and one end node.
- With this graph, statement coverage is the ratio of all visited nodes to all nodes. Full statement coverage isn't sufficient for thorough testing.
- Branch coverage is the ratio of all visited edges between nodes in the CFG to all edges. This insufficiently tests loops.
- Path coverage is the ratio of all visited paths to all paths, where a path is any sequence of edges from the start to the end node. The problem is that with loops, there can be an infinite number of paths, so full path coverage can't be practically tested.
It is therefore often useful to check condition coverage.
- In simple condition coverage, each atomic condition is once true and once false – but this doesn't guarantee full statement coverage.
- In multiple condition coverage, the atomic conditions have taken on all combinations of
true and false. This implies full branch coverage, but is rather expensive. The program may have additional constraints that exclude certain combinations. This technique is good for obtaining branch coverage, can find dead code, but cannot find bugs stemming from the wrong condition.
- In Minimal multiple condition coverage, each atomic and composite condition is once true and false. It still implies full branch coverage. It is a subset of multiple condition coverage, but requires fewer test cases.
When constructing test input using condition coverage, then short-circuiting should be taken into account. For example,
function foo(A, B) {
if (A && B) x()
else y()
}
needs to be tested with foo(false, whatever), foo(true, false), and foo(true, true) for full minimal multiple condition coverage.
If you have objects that can be in multiple states, then testing all state transitions analogous to control flows seems sensible.
There are some more complex coverage metrics, but they are generally similar to the metrics presented here.
These are white box testing methods, and can be partially automated. Note that an unit test suite should aim to have a high code coverage by any chosen metric, but 100% isn't always possible. It is especially difficult to test exception handling, where faults have to be injected into specific locations.
Functional tests
Then there are functional tests which assert that the code adheres to the spec by viewing the implementation as a black box. Such tests are useful for unit tests and integration tests alike. Because it is impossible to test with all possible input data (e.g. testing string length with all possible strings), it is useful to group the input (and output) into equivalent classes - if length("foo") is correct, foo("bar") is likely to work as well. For each possible combination between input and output equivalence classes, at least one representative input is chosen and tested.
One should additionally test
- edge cases
length(""), foo("x"), length(longer_than_INT_MAX),
- values that are permitted by the language, but not by the contract of the function
length(null), and
- possible junk data
length("null byte in \x00 the middle")…
With numerics, this means testing 0, ±1, ±x, MAX, MIN, ±∞, NaN, and with floating point comparisons testing two neighboring floats. As another addition, random test values can be picked from the equivalence classes. To ease debugging, it is worth recording the seed used…
Non-functional tests: Load tests, Stress tests
A piece of software has non-functional requirements, which have to be tested as well. These include testing at the defined boundaries (load tests), and beyond them (stress tests). For a computer game, this could be asserting a minimum number of frames per second in a load test. A website may be stress-tested to observe response times when twice as many visitors as anticipated are battering the servers. Such tests are not only relevant for whole systems but also for single entities – how does a hash table degrade with a million entries?
Other kinds of tests are whole system tests where scenarios are simulated, or acceptance tests to prove that the development contract was fulfilled.
Non-testing methods
Reviews
There are non-testing techniques that can be used for quality assurance. Examples are walkthroughs, formal code reviews, or pair programming. While some parts can be automated (e.g. by using linters), these are generally time intensive. However, code reviews by experienced programmers have a high rate of bug discovery, and are especially valuable during design, where no automated testing is possible.
When code reviews are so great, why do we still write tests? The big advantage of test suites is that they can run (mostly) automatically, and are as such very useful for regression tests.
Formal verification
Formal verification goes and proves certain properties of the code. Manual verification is mostly viable for critical parts, less so for whole programs. Proofs put a lower bound on program correctness. Proofs can be automated to a certain degree, e.g. via a static type checker.
Certain invariants can be explicitly checked by using assert statements.
All of these techniques have their place, and are complementary. TDD writes the functional tests up front, but the tests can be judged by their coverage metrics once the code is implemented.
Writing testable code means writing small code units that can be tested separately (helper functions with suitable granularity, single responsibility principle). The fewer arguments each function takes, the better. Such code also lends itself for insertion of mock objects, e.g. via dependency injection.
double pihole(double value) { return (value - Math.PI) / (value - Math.PI); }which I learned from my math teacher. This code has exactly one hole, which can't be discovered automatically from black-box testing alone. In Math there is no such hole. In calculus you're allowed to close the hole if the one-sided limits are equal.(a,b)=>a/b, which needs to be extended with an overflow value, in order to be correctly composable.