31

Is it feasible to expect 100% code coverage in heavy jquery/backbonejs web applications? Is it reasonable to fail a sprint due to 100% coverage not being met when actual code coverage hovers around 92%-95% in javascript/jquery?

9
  • 8
    “Fail a sprint” sounds weirdly ominous… Jun 14, 2012 at 20:59
  • 5
    It's an asymptote. Jun 14, 2012 at 21:18
  • 13
    even if you have full coverage some bugs won't be found so don't rely on that number to fix everything Jun 15, 2012 at 0:53
  • 12
    Anything is possible. The real question is whether the value of 100% code coverage is worth the cost in time and resources.
    – JohnFx
    Jun 15, 2012 at 3:45
  • 5
    Why are you worrying about this, when the underlying assumption--that 100% (or any other number) automated test coverage will magically make your code better--is a pipe dream itself? Jun 15, 2012 at 22:41

13 Answers 13

33

It is equally realistic as it is unrealistic.

Realistic
If you have automated testing that has been shown to cover the entire code base, then insisting upon 100% coverage is reasonable.
It also depends upon how critical the project is. The more critical, the more reasonable to expect / demand complete code coverage.
It's easier to do this for smaller to medium sized projects.

Unrealistic
You're starting at 0% coverage ...
The project is monstrous with many, many error paths that are difficult to recreate or trigger.
Management is unwilling to commit / invest to make sure the coverage is there.

I've worked the gamut of projects ranging from no coverage to decent. Never a project with 100%, but there were certainly times I wished we had closer to 100% coverage.
Ultimately the question is if the existing coverage meets enough of the required cases for the team to be comfortable in shipping the product.

We don't know the impact of a failure on your project, so we can't say if 92% or 95% is enough, or if that 100% is really required. Or for that matter, the 100% fully tests everything you expect it to.

15
  • 32
    ... And just because you have 100% code coverage doesn't mean you have 100% branch coverage, so even with 100% code coverage you could be missing a lot. Jun 14, 2012 at 23:08
  • 4
    +1 for project size. Breaking down into smaller, reusable, and testable components has allowed us to gain ~95% coverage ourselves. 100% coverage is not necessary. Integration testing should cover unit testing gaps.
    – Apoorv
    Jun 14, 2012 at 23:11
  • 5
    @BryanOakley ...and also your tests could be pointless, or not even test anything
    – David_001
    Jun 15, 2012 at 13:58
  • 6
    @BryanOakley And even with 100% branch coverage, it's possible that a certain combination of branches can cause a problem. (two sequential IF statements, for example, can be branched into and around in separate tests, but missing a test that enters both. Full branch coverage, yet one execution path is missed)
    – Izkata
    Jun 15, 2012 at 19:55
  • 5
    Even 100% branch coverage, including all execution paths is not enough. Maybe some error only happens when you take some combination of branches and you have some external input, say a malformed date. There is no possibility that all cases will ever be covered. At the same time, one can have a good confidence with less than 100% coverage but suitably chosen edge cases as input.
    – Andrea
    Jun 16, 2012 at 15:55
33

Who tests the tests?

It is very naive at best and unrealistic even in the theoretical sense and impractical in a business sense.

  • It is unrealistic with code that has high cyclomatic complexity. There are too many variables to cover every combination.
  • It is unrealistic with code that is heavily concurrent. The code is not deterministic so you can't cover every condition that might happen because behavior will change on every test run.
  • It is unrealistic in a business sense, it only really pays dividends to write tests for code that is critical path code, that is code that is important and code that may change frequently.

Testing every line of code isn't a good goal

It is very expensive to write tests, it is code that has to be written and tested it self, it is code that has to be documented in what it actually trying to test, it is code that has to be maintained with business logic changes and the tests fail because they are out of date. Maintaining automated tests and the documentation about them can be more expensive than maintaining the code sometimes.

This is not to say that unit test and integration tests aren't useful, but only where they make sense, and outside of industries that can kill people it doesn't make sense to try and test every line of code in a code base. Outside these critical kill lots of people quickly code bases, it is impossible to calculate a positive return on investment that 100% code coverage would entail.

Halting problem:

In computability theory, the halting problem is the problem of determining, from a description of an arbitrary computer program and an input, whether the program will finish running or continue to run forever.

Alan Turing proved in 1936 that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist. A key part of the proof was a mathematical definition of a computer and program, which became known as a Turing machine; the halting problem is undecidable over Turing machines. It is one of the first examples of a decision problem.

Since you can not even prove something works 100% why make that your goal?

Plain and simple, in most cases it doesn't make any business sense.

4
  • 8
    this really needs to be the accepted answer. 100% code coverage is almost as bad as 0%.
    – Ryathal
    Jun 15, 2012 at 20:30
  • 2
    "There are too many variables to cover every combination." This has nothing to do with getting 100% code coverage. If a line of code was important enough to be written, and it is important enough to keep around, then it is important enough to be covered by a test. If it is not covered by a test, the only safe assumption is it doesn't work. It is true that for some code it does not make sense from a business perspective to test it. That is the very same code that did not make sense from a business perspective to write. Aug 7, 2017 at 2:20
  • 2
    so you think writing test cases to cover simple getXXX()/setXXX() sor simple assignment constructors for value objects are a good use of time and resources, sorry that is just not the case in reality and an extremely naive opinion that lacks real world experience to back it up. Remember test code is still code that has to be maintained. The less code you write to solve a problem the better in every case.
    – user7519
    Aug 7, 2017 at 3:58
  • 1
    Uhm "It is unrealistic with code that has high cyclomatic complexity. There are too many variables to cover every combination." - of course, that's why you have to break such code into smaller pieces which have small cyclomatic complexity and are thus easier to test. Refactoring in that way is essential for testing - it makes testing easier. Nov 22, 2018 at 16:16
17

In most cases, 100% code coverage means that you've "cheated" a little bit:

  • Complex, frequently changing parts of the system (like the gui) have been moved to declarative templates or other DSLs.
  • All code touching external systems has been isolated or handled by libraries.
  • The same goes for any other dependency, particularly the ones requiring side effects.

Basically, the difficult to test parts have been shunted to areas where they don't necessarily count as "code". It's not always realistic, but note that independent of helping you test, all of these practices make your codebase easier to work on.

2
  • How is moving things to DSLs cheating?
    – back2dos
    Jun 18, 2012 at 1:07
  • 2
    @back2dos - While you might unit test say, your embedded python scripts, you likely aren't unit testing your html templates, or your CSS, or counting the lines in them towards towards coverage estimates.
    – Dan Monego
    Jun 18, 2012 at 13:16
12

For an impressive, real world example of 100% branch coverage, see How SQLite is Tested.

I realize your question specifically asks about javascript which is an entirely different type of software product, but I want to bring awareness to what can be done with sufficient motivation.

10

100% code coverage for unit tests for all pieces of a particular application is a pipe dream, even with new projects. I wish it were the case, but sometimes you just cannot cover a piece of code, no matter how hard you try to abstract away external dependencies. For example, let's say your code has to invoke a web service. You can hide the web service calls behind a interface so you can mock the that piece, and test the business logic before and after the web service. But the actual piece that needs to invoke the web service cannot be unit tested (very well anyway). Another example is if you need to connect to a TCP server. You can hide the code that connects to a TCP server behind a interface. But the code that physically connects to a TCP server cannot be unit tested, because if it is down for any reason then that would cause the unit test to fail. And unit tests should always pass, no matter when they are invoked.

A good rule of thumb is all of your business logic should have 100% code coverage. But the pieces that have to invoke external components, it should have as close to 100% code coverage as possible. If you cannot reach then I wouldn't sweat it too much.

Much more important, are the tests correct? Do they accurately reflect your business and the requirements? Having code coverage just to have code coverage doesn't mean anything if all you doing is testing incorrectly, or testing incorrect code. That being said, if your tests are good, then having 92-95% coverage is outstanding.

3
  • 1
    Testing what happens when you get strange combinations of error cases and failure-to-responds can be exceptionally tricky. Jun 14, 2012 at 21:33
  • Isn't understanding what your system will do when presented with these tricky problems part of the appeal of unit testing? Also, there's a bit of confusion here between unit tests and integration tests. Jun 15, 2012 at 0:38
  • this conflates unit testing with integration testing, testing code you did not write is integration testing. The TCP stack is in the OS you should not be testing that, you should assume it is already tested by whom ever wrote it.
    – user7519
    Apr 30, 2015 at 17:23
5

I'd say unless the code is designed with specific goal of allowing 100% test coverage, 100% may be not achievable. One of the reasons would be that if you code defensively - which you should - you should have sometimes code that handles situations that you're sure shouldn't be happening or can't be happening given your knowledge of the system. To cover such code with tests would be very hard by definition. To not have such code may be dangerous - what if you're wrong and this situation does happen one time out of 256? What if there's a change in unrelated place which makes impossible thing possible? Etc. So 100% may be rather hard to reach by "natural" means - e.g., if you have code that allocates memory and you have code that checks if it has failed, unless you mock out memory manager (which may not be easy) and write a test that returns "out of memory", covering that code might be difficult. For JS application, it may be defensive coding around possible DOM quirks in different browsers, possible failures of external services, etc.

So I would say one should strive for being as close to 100% as possible and have a good reason for the delta, but I would not see not getting exactly 100% as necessarily failure. 95% can be fine on a big project, depending on what the 5% are.

1
  • Just because the code is not supposed to be run in production under normal circumstances does not mean it can't be written in such a way as to be run by the tests. How critical is it for that unusual code to run correctly? Is it important enough to cover it by tests? I would argue that if it is not important enough to cover by tests, it is not important enough to handle that case. Code that doesn't need tests is code that doesn't need to exist, and should be deleted. Aug 7, 2017 at 3:22
2

If you are starting out with a new project, and you are strictly using a test-first methodology, then it is entirely reasonable to have 100% code coverage in the sense that all of your code will be invoked at some point when your tests have been executed. You may not however have explicitly tested every individual method or algorithm directly due to method visibility, and in some cases you may not have tested some methods even indirectly.

Getting 100% of your code tested is potentially a costly exercise, particularly if you haven't designed your system to allow you to achieve this goal, and if you are focusing your design efforts on testability, you are probably not giving enough of your attention to designing your application to meet it's specific requirements, particularly where the project is a large one. I'm sorry, but you simply can't have it both ways without something being compromised.

If you are introducing tests to an existing project where testing has not been maintained or included before, then it is impossible to get 100% code coverage without the costs of the exercise outweighing the effort. The best you can hope fore is to provide test coverage for the critical sections of code that are called the most.

Is it reasonable to fail a sprint due to 100% coverage not being met when actual code coverage hovers around 92%-95% in javascript/jquery?

In most cases I would say that it you should only consider your sprint to have 'failed' if you haven't met your goals. Actually, I prefer not to think of sprints as failing in such cases because you need to be able to learn from sprints that don't meet expectations in order to get your planning right the next time you define a sprint. Regardless, I don't think it's reasonable to consider the code coverage to be a factor in the relative success of a sprint. Your aim should be to do just enough to get everything to work as specified, and if you are coding test-first, then you should be able to feel confident that your tests will support this aim. Any additional testing you feel you may need to add is effectively sugar-coating and thus an added expense which can hold you up in completing your sprints satisfactorily.

4
  • "I'm sorry, but you simply can't have it both ways without something being compromised." That is not true. You can always scale back the features, or go slower. If something is not worth testing it is not worth writing. If a line of code is important enough to keep around, it is important enough to test. If it is not important enough to test, it is not important enough to keep around. The only safe assumption of a line of code that is not tested is that it does not work. Aug 7, 2017 at 2:27
  • @still_dreaming_1, you seem to have supported my statement and contradicted yourself. Scaling back features or altering your deadlines are compromises, each of which can affect project cost and stakeholder expectations. Testing legacy code that hasn't previously been tested fully is extremely difficult, as you have to understand not only the code as it runs, but the intentions of the original creator, and writing tests that capture existing legacy code behaviour doesn't necessarily show that the code works entirely as it is supposed to.
    – S.Robins
    Sep 12, 2018 at 5:19
  • I guess my point was that the something that was compromised, the features or changes that didn't get created yet because development is moving faster, is not a real compromise because if you lose coverage in order to move faster, those feature and changes can only be assumed to not work right anyways. So then what was the point of making those changes or adding those features if it doesn't matter whether they work right or not? If it doesn't matter whether or not they work right, those changes didn't need to be made, and should now be yanked out of the code. Sep 14, 2018 at 13:39
  • I don't completely believe that anymore, or at least I realize the practicality aspect of the truth you are speaking, especially in a legacy code base, so that is just an explanation of the point I was trying to make at the time. Actually, I'm now completely conflicted about even doing TDD all the time on a new code base, let alone getting 100% coverage. On the one hand, every form of logic and reason tells me both of those things should be good, and yet in practice I can't seem to make it practical. So something is very wrong with the programming world, we need a new paradigm. Sep 14, 2018 at 13:41
1

I don't do this as a matter of course, but I have done it on two large projects. If you've got a framework for unit tests set up anyway, it's not hard exactly, but it does add up to a lot of tests.

Is there some particular obstacle you are encountering that is preventing you from hitting those last few lines? If not, if getting from 95% to 100% coverage is straightforward, so you might as well go do it. Since you're here asking, I'm going to assume that there is something. What is that something?

1
  • This is one of the best answers here. Asking what is preventing a line of code from being easily coverable is a good question. Getting those lines covered will force you to improve the code to make it happen, so it will be a win, win. Aug 7, 2017 at 2:29
1

Martin Fowler writes in his blog: I would be suspicious of anything like 100% - it would smell of someone writing tests to make the coverage numbers happy, but not thinking about what they are doing.

However, there are even standards that mandate 100% coverage at unit level. For example, it is one of the requirements in the standards of the European spaceflight community (ECSS, European Cooperation for Space Standardisation). The paper linked here, tells an interesting story of project that had the goal of reaching 100% test coverage in an already completed software. It is based on nterviews with the involved engineers who developed the unit tests.

Some of the lessons are:

  • 100% coverage is unusual but achievable
  • 100% coverage is sometimes necessary
  • 100% coverage brings in new risks
  • Don’t optimize for the 100%-metric
  • Develop a proper strategy to maximize coverage
  • 100% coverage is not a sufficient condition for good quality
0

92% is fine. I feel that the real questions are:

  • Is 92% the 'new' norm now? If the next sprint has 88% testing, will that be ok? This is frequently the start of test suites being abandoned.

  • How important is it that the software work and not have bugs. You have tests for these reasons, not "for the sake of testing"

  • Is there a plan to go back and fill in the missing tests?

  • Why are you testing? It seems like the focus is % of line covered not functionality

1
  • "How important is it that the software work and not have bugs"? Good question. What is the definition of a bug? Something that does not work as intended. If it is ok for some code to not work correctly then don't write it. The whole point of code is for it to work. Aug 7, 2017 at 2:30
0

Perhaps asking if is feasible and reasonable are not the most helpful questions to ask. Probably the most practical answer is the accepted one. I will analyze this on a more philosophical level.

100% coverage would be ideal, but ideally, it would not be needed, or would be much easier to achieve. I prefer to think about if it is natural and human than feasible or reasonable.

The act of programming correctly is next to impossible with today's tools. It is very difficult to write code that is totally correct, and doesn't have bugs. It is just not natural. So, with no other obvious option, we turn to techniques like TDD, and tracking code coverage. But as long as the end result is still an unnatural process, you will have a hard time getting people to do it consistently and happily.

Achieving 100% code coverage is an unnatural act. For most people, forcing them to do achieve it would be a form of torture.

We need processes, tools, languages, and code that map to our natural mental models. If we fail to do this, there is no way to test quality into a product.

Just look at all the software out there today. Most of it messes up pretty regularly. We don't want to believe this. We want to believe our technology is magical and make us happy. And so we choose to ignore, excuse, and forget most of the times our technology messes up. But if we take an honest appraisal of things, most of the software out there today is pretty crappy.

Here are a couple efforts to make coding more natural:

https://github.com/jcoplien/trygve

https://github.com/still-dreaming-1/PurposefulPhp

The later is extremely incomplete and experimental. Actually it is a project I started, but I believe it would be a huge step forward for the craft of programming if I could ever get myself to put the time into it to complete it. Basically it is the idea that if contracts express the only aspects of a classes behavior that we care about, and we are already expressing contracts as code, why not only have the class and method definitions along with the contracts. In that way the contracts would be the code, and we would not need to implement all the methods. Let the library figure out how to honor the contracts for us.

-2

Reaching 100% on new code should be very achievable and if you're practicing TDD you'll likely hit that by default as you're very deliberately writing tests for every line of production code.

On existing legacy code that was written with no unit tests it can be difficult as often legacy code wasn't written with unit testing in mind and can require a lot of refactoring. That level of refactoring often isn't practical given the realities of risk and schedule so you make trade offs.

On my team I specify 100% code coverage and if we see less than that in the code review the technical owner of the component discuss why 100% wasn't reached with the developer and must agree with the developer's reasoning. Often if there's a problem hitting 100% the developer will talk to the technical owner before the code review. We've found that once you get into the habit and learn techniques for working around several common issues with adding tests to legacy code that hitting 100% regularly is not as difficult as you'd initially think.

Michael Feather's book "Working Effectively with Legacy Code" has been invaluable to us for coming up with strategies for adding tests to our legacy code.

0
-3

No, it's not possible and it never will be. If it were possible all of mathematics would fall into finitism. For example, how would you test a function that took two 64 bit integers and multiplied them? This has always been my problem with testing versus proving a program correct. For anything but the most trivial programs, testing is basically useless as it only covers a small number of cases. It's like checking a 1,000 numbers and saying you've proved the Goldbach conjecture.

2
  • Oh! So somebody is upset that I didn’t answer the problem on the plane of its conception; testing is a waste… I don’t care if it’s popular. It will never work. It cannot. The smartest computer scientists have known this (Dijkstra, Knuth, Hoare et al.). I guess if you’re a JavaScript programmer huffing eXtreme Programming then you don’t care about those cranks, though. Blah, whatever, who cares… write crappy code. Waste CO^2 running your tests. — I mean, who has time to sit and think anymore? We’ve exported that to the computer. Jun 17, 2012 at 17:22
  • 3
    The question is tagged "TDD". TDD is more a design tool and a problem exploration tool than a testing one, and each "test" is just an example of how the code will behave in some context, so that people can read and understand what's going on, then change it safely. TDD done well tends to lead to cleaner, easier-to-use code, and running the tests just checks that the documentation is current. Most TDD suites hardly ever catch bugs; it's not what they're there for. I think you're being downvoted because your answer betrays that lack of understanding, and I hope this comment helps with that.
    – Lunivore
    Aug 20, 2012 at 13:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.