4

In thinking about formal verification techniques, you also need to have test coverage. But as that article shows, you can easily say you have "100% test coverage", but every test doesn't have any assertions. That is the extreme. But I'm wondering if there is ever a situation where you can say that you have 100% test coverage somehow. Similar to how a model checker can check every possible path and state in your application, wondering if a testing tool can be said to cover the same level of thoroughness as a model checker. Basically wondering if there is a way to be thorough enough with your tests to know that you have 100% percent coverage, without explicitly writing every possible combination of things.

  • 1
    Yes, the tool is "write test first". Write production code only when you have failing test for the logic you want to implement. – Fabio Jun 29 '18 at 5:40
  • Fowler uses "test coverage" = "code coverage". So if want to know if you have 100% percent code coverage, or not, use a coverage tool. Still you will have to design your tests to achieve this, but that is not what you literally asked for. – Doc Brown Jun 29 '18 at 10:08
  • 1
    You might find this to be interesting reading: How SQLite Is Tested. SQLITE has 100% branch coverage, and yet, bugs still crop up from time to time). – Bryan Oakley Jun 29 '18 at 14:02
  • 1
    Formal verification is usually more about automated proofs, not tests. – Frank Hileman Jun 29 '18 at 16:01
11

There are strategies that can produce deep code coverage metrics. Mutation testing is one example. Roughly speaking, mutation testing ensures that any logical change to the program results in a failed test by actually running the tests against every logical permutation of a program. If a logical change doesn't produce a failing test, it represents a meaningful test coverage gap.


My disclaimer: I've never actually performed mutation testing. I can't speak to how feasible or valuable it really is. But, in theory, it sounds pretty darn slick. In practice, however, most businesses can survive (and thrive) on hiring good developers who are engaged, interested, and insist on following good development practices.

  • 1
    Calculating all the permutations possibles of a program might well have a cost in terms of efforts, time and money. What should make us think whether a 100% coverage worth such costs. Sure there're situations where it worth it. Even more, it should be possible and it should be achieved. However, in mainstream developments, I have seen none. I put a lot of efforts into having covered at least 85% of my code but it never guaranteed me a program free of bugs. It leads me to realise that there're other metrics that matter more than coverage. Like complexity and technical debt. – Laiv Jun 29 '18 at 8:49
  • @Laiv though what's smart about the mutation testing strategy (as I understand it) is that it doesn't aim to generate test cases for all permutations. What it does is when you change a part of code, it generates a single test case which causes different program states for the code before and after the change. And then it checks if this generated test case is still passing or not, which I think is pretty nifty. – Peeyush Kushwaha Jun 30 '18 at 13:19
  • Yes. It sounds interesting. As soon as its development and implementation doesn't take you too much "time". – Laiv Jun 30 '18 at 14:17
8

100% test coverage is possible, and despite 100% coverage, your program may still have errors. See shortcomings of test coverage at "Does path coverage guarantee finding all bugs?"

Similar to how a model checker can check every possible path and state in your application, wondering if a testing tool can be said to cover the same level of thoroughness as a model checker.

The basic idea is that test coverage will only measure if you've been through a particular statement / path etc or not, at-least in one state (in contrast to model-checking which provides guarantees for all possible states)

So the answer is no, unless you have test cases for every possible input-output pair (exponentially large number of test cases in the general case)

2

You can have 100% coverage, but the question is: does it mean anything? Personally, I would rather have 60% coverage of the important methods and their scenarios, than 100% coverage of non-relevant scenarios.

1

Depends my complexity of what you're testing. When my application only prints Hello World I only have to run an assert to check if the printed result is "Hello World". When real time performance issues, hardware limitations are involved, or there are very complex interactions I imagine some test cases that no one could expect can occur.

-1

Yes, 100% Test coverage is possible. It always varies from application to application, the major factors that ensures test coverage are size of the application, complexity of the code and project bandwidth. Small the size of the application and more the coverage is achievable.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.