Testing : Why is it necessary?

Question

Hello I will try to explain what is my actual understanding of tests and why I have problems to grasp it's utility.

Even if I try my hardest to understand the concept of TTD, unit test, integration test, etc. From the most articles that I read it explain HOW to do it (with a trivial example or a framework) and sometimes how the tests saved them hours of debug in certain situation. And I have two problems with that :

First : the method

Let's use a basic example for the explaination of how to do a unit test :

You want to test that your add function works as expected so in your test file you write a mock of your function with an expected output lets say something like that :

import add from ./calculate
function myTestingFunction() {
    add(3, 5)
    expectedOutput(8)
}

Okay so you explain me how to do it but not why... and even with that, I have multiple questions :

• You're testing a function with another function ... so why did you stop there and test the function you just created ?
• It's code, it has algorithm so how to be sure it is correctly written ?
• And why stop with the expected output ? Why not testing that the input are of the correct type, because if I give characters to my add function it will fail, no ?

And every article is basically like that. So with that questions in mind I cannot find the motivation to even try this methods : it's just don't seems logic.

Second : Tests are time and money saving

Let's say now I want to test the Divide function the same way and write and expected output of 3 for the input 9 and 3.

Okay so I write the same function than explained before. But if the user tries to divide by 0 a serious bug will logically occur for him ... this test doesn't covers this case and I'm writing test to ensure my code works well (or at least give the correct outputs).

In real life I now see two situation :

• I had explained to my client that I will take more time to code (so he will pay me more) but as a result he will be sure that in final his functions are bullet proof. And here I'm saying to him :

"Well ... sorry I didn't think about this use case"

"So I pay you more because you said that the tests you've written will ensure a bug free app... but it's not ? And now I have to pay you to debug it ?"

(so no time and money save and a bad relation with my client)

• I had coded the functionnality as explained and understood by me and my client. The new use case occur that we were'nt aware of. 'Kay no problem : I pull a branch to bug fix, fix the bug, push the hotfix, document the special use case if needed. And now in the future if I encounter an equivalent problematic I will have the algorithm already prepared because I faced the issue (time and money save for me and my client and a better relation overall)

I don't know if I'm clear about my state of mind but I've discussed these elements with many of my colleagues (some even with more years of practice or a better school degree than me) and they had the same problem than me about test : even if we really want to understand this topic and implement a good way of working if needed, as of now i'ts seems like some sort of intelectual pride.

UPDATE

Thank you all for your answers and comments, I learned a lot of terminology that will help my future research in order to clear my mind.

I also want to mark a new point :

How to make a relevant test

It seems you assume I know how to make a good test set to prove a point of my code or to validate my understanding of a function before pushing it to prod or anything.

It's not the case : even if I know how to do it regarding syntax to use or framework to implement let's say. I didn't find any resources to learn what is considered a "good" test set or a "right" way of doing it. That's why I don't know where do I stop the scenario of testing or how many expecting outputs it's need, if it is a question of quantity of output as well or if can just narrow to the not expected behavior maybe :

unexpectedOutput(!= 8) if my output differ from 8 my test fail

I didn't find resources to this very root knowledge about test and it seems to be a complete different job than the web developper that I am.

The same way I can configure let's say a CI/CD pipeline and implement several tools around it to automate it. But I'm not a devops, there are surely errors of optimisations I made or another tool more dedicated to the one I used but I didn't learn about it, etc.

Answer 1

Testing is not necessary; it is inevitable

Your software will get tested. If not by you, then by your customers. You are not required to test, but it is kind to your customers.

The earlier the test, the cheaper the fix

When a customer finds a bug, he has to spend time figuring out if he is doing something wrong, look for workarounds on the internet, then log the bug with your company. Your company then has to assign an engineer, work the bug into the product roadmap, and create a release. All that is really expensive, and takes a long time. So it would be better to find the bug yourself.

If your find your bug during QA, the QA engineer has to determine the steps to reproduce, report the bug, triage it, enter it into the system, and it has to go through your SDLC. That is cheaper than a customer bug but still a lot of work.

If your developer finds the bug before QA, he just fixes it. It is far far less expensive to do it then.

Testing the tests

As you have figured out, the green light you see on the testing console does not tell you if the code is correct, since the test itself could be wrong. But that is not what the green light is supposed to tell you-- it has no way of knowing what "correct" means. Instead, what the green light does tell you is that the test agrees with the code. And that means it can tell you, at any given moment, what parts of the system have changed.

Why is that valuable? I don't think it's because it helps you write the first draft of the code. Indeed, it doubles the work. But it does help you with the second, third, and fourth drafts, and so on, throughout the product's life cycle. It helps because it tells you when you've accidentally broken something, meaning you can work faster and with more confidence without compromising on quality. Also, it makes it easier to let other developers work on your code base, since you can review the test results when approving their pull request.

Note: There are some purists who will tell you that you should write the tests first. That would seem to imply that the tests are meant to help you with that first draft of code, and they do, but that is because of the way guides the engineer's thought process and forces them to think about requirements first. At that point the benefit comes from the process, not the test itself. And that is good too.

Answer 2

Testing saves time and money largely because we're not as good at abstract thinking as we think we are.

Personally, I have a PhD in computer science, am a team lead and have worked on maybe 100 projects of more than trivial size. I pride myself that I understand making the computer do desired things better than the average, and the metrics confirm this. But as soon as a system grows beyond a surprisingly small size, I'm no longer able to predict its behaviour well enough to avoid errors, and specifying it in advance through explicit tests becomes a net time saver..

I suspect that the trouble with most advocacy for TDD is that they use small bits of code so that they can be read quickly and understood easily. But that is exactly the kind of code that needs it least.

If you allow me to brag a second time, I'm reasonably certain that I can slam out getters/setters for a data class and maybe a simple wrapper for arithmetic on numbers without errors. But anything above that becomes hard because virtually always, different parts of a specification interact with each other. Assume that you have just four dimensions along which a task may vary, and each allows three alternatives. You now have 81 possible combinations of circumstances, and that is way too many to keep in your head. Many of them will be harmless; some will be unexpected; some will expose gaps in the spec (what does it mean to divide by zero? To raise a negative number to a fractional power? To arrive at a number that is smaller than your possible resolution, but definitely different from zero?); some will force you to rethink large parts of the solution, or even the problem.

This seems to be a surprisingly constant phenomenon. I've worked with and trained a lot of programmers; I have never, ever, met one who understood things as well as they initially thought. That is why fixating expected outcomes in a machine-verifiable way is a good idea, and that's basically what testing does. It verifies not only that you have a clear grasp of what is supposed to happen in a situation, but also that you've reached that goal, and that you don't backslide from that achievement through later changes.

There is a famous quotation by Donald Knuth saying something like, "Beware of this code; I have only proved it correct, not actually tested it". My attitude is that unless you are smarter than Donald Knuth, you need tests for any nontrivial code base, and it has served me well.

Answer 3

First : the method

[...]

Okay so you explain me how to do it but not why... and even with that, I have multiple questions :

• You're testing a function with another function ... so why did you stop there and test the function you just created ?
• It's code, it has algorithm so how to be sure it is correctly written ?

The idea with writing tests is that the test code is simple enough that you can read the code and immediately see if it does what it is supposed to do. A typical test function is just a linear sequence of statements with as most complex feature a loop.

On the other hand, the code being tested is typically way more complex and hard to fully understand for the average programmer.

• And why stop with the expected output ? Why not testing that the input are of the correct type, because if I give characters to my add function it will fail, no ?

You don't stop with a test for the expected output. The more interesting tests are for corner cases in the inputs and unexpected inputs. For any non-trivial function, you should expect to have a multitude of test cases.

The problems with most examples used in blogs and tutorials is that they are so simple that the functions you are testing are (almost) as simple as the test code itself and can be verified for correctness without an extensive test suite.

I had explained to my client that I will take more time to code (so he will pay me more) but as a result he will be sure that in final his functions are bullet proof.

Tests don't make your code bullet proof. They provide a proof of the situations you have thought of for proving that your production code works. Without tests, how does your client know that your code does what was requested and that it is robust against most incorrect usage scenarios?

Answer 4

I had coded the functionality as explained and understood by me and my client. The new use case occur that we weren't aware of. 'Kay no problem

If you're lucky it's not a problem. If you're not lucky, it is a problem, and someone has experienced financial loss, time lost, or for some applications serious injury or death. I do think every software developer should read a bit of RISKS and its enormous catalogue of disasters, large and small. Not too much or you'll never want to write software again.

Some companies, including very prestigious ones, can afford to "test in production" because they have very good systems for spotting problems and rolling back the effects. Most companies have to determine that something works to an adequate standard before shipping it.

A good document on high-standard testing is that of SQLite:

It is (relatively) easy to build an SQL database engine that behaves correctly on well-formed inputs on a fully functional computer. It is more difficult to build a system that responds sanely to invalid inputs and continues to function following system malfunctions. The anomaly tests are designed to verify the latter behavior.

The scope of possible inputs, outputs, and adverse circumstances is huge. It is often difficult to reason about.

Back to your question:

But if the user tries to divide by 0 a serious bug will logically occur for him ... this test doesn't covers this case

Well, yes. That means your test suite is inadequate and you need more tests. This is where the question of test coverage arises; automated tools can tell you how much of your code is actually run during the tests and thereby suggesting extra tests to add. Experienced human testers are good at guessing what sort of inputs are likely to cause problems. Zero, negative numbers, giant numbers, long strings, invisible characters, weird bits of UNICODE, and so on.

Answer 5

Tests of all sorts don’t guarantee much of anything. They are risk mitigation. As a human, you will make mistakes. Tests exist to catch those mistakes. Yes, the odds of making a mistake are low, but the odds of making two mistakes (one in the code and one in the test) multiply together. Two 1 in 20 chances becomes 1 in 400. If 1 in 400 is still too risky, add more tests.

Yes, you still need to do a good job writing the tests to cover input checking, bounds checking, etc. And even if the code is perfect, it might not solve the problem at hand. But catching mistakes early (while the code is still fresh in your mind, before it produces bad data, before other programmers depend on it) generally makes them cheaper to fix than without tests.

Answer 6

The cost of writing software

Consider any new 'green field' project which starts from a clean slate; chances are that its developers probably had to produce MVP within a sensible timeframe (Perhaps a year or maybe two at most); many projects often start out with just one small (e.g. "two-pizza") team, or even smaller.

The problem is that this point in a software's overall lifespan is just MVP - software typically isn't just built and just left alone for years; indeed the term soft was originally used to imply that (unlike hardware), it should be easy to change and easily malleable, and stakeholders typically expect and demand it to be easily changed.

Whatever costs were incurred to reach MVP with an initial small team and limited set of simple requirements is often tiny compared to later costs as they accumulate over subsequent years as the product grows; requirements gain more complexity; a codebase has ever-growing moving parts with more and more developers contributing over time, and as the distant memories of the original requirements growing ever-fainter.

The cost of maintaining software

Consider software which reached MVP even 3 years ago; the original team may already have all left their jobs and moved elsewhere, but the stakeholders who paid for it likely still need the software and are also probably highly dependent upon its continued stability and need it to keep evolving and growing in scope.

New developers can't realistically know the history of the code nor its design, nor its original requirements, nor all the decisions or trade-offs which took place in its lifetime, nor all the "gotchas" and other quirks or hidden bits of tech debt that nobody ever had time to fix.

With knowledge of the software and all the decisions being lost (or buried in obscure wikis and other decaying documentation that nobody knows how to navigate) and new developers essentially inheriting legacy code, each time a change happens which impacts the existing code, developers should rightly be expected by stakeholders not to have broken that code, and to release something whose quality is equal to or better than the previous version.

If the original developers covered their code comprehensively in automated tests; the quality is easy to assure because those hundreds (maybe thousands) of tests should run in just a few minutes and highlight the regression errors before anyone even creates a Merge Request. The CI pipeline can immediately warn the current developers when some long-lost requirement in the tests somewhere has gotten broken and prevent the broken functionality being merged into the main branch.

If, on the other hand, the original code was written without tests; consider how many opportunities the intervening years might have created to break that original code and introduce regression issues which a manual tester might miss and could easily end up making it into production or be unnoticed for a long time.

The reality is that code-entropy can grow quickly -- a single change to a single line of code can create a bug; if developers are constantly changing tens/hundreds of lines of code or refactoring and adding/changing/removing logic linked to existing code, then the opportunities to break existing code grows exponentially.

The cost of testing software

while opportunities to break existing code only ever grows, a team's ability to detect those bugs by-hand diminishes; the knowledge gets lost and fades over time, people forget things, the huge increase in scope makes comprehensive manual testing increasingly expensive and less feasible as more code is added and the software deals with an ever-expanding list of increasingly complex requirements.

Testing is the only real protection any team has against existing code getting broken; and there are only two choices - either make it happen automatically, or pay someone to do it manually.

Writing automated tests incurs an up-front cost once; involving the original developer and at a point in time when everyone is fully aware of the requirements and expectations, so everyone has the knowledge fresh in their head.

Writing manual tests incurs repeated costs which grow over time; initially the original QA tester is likely to have been involved in the original discussion around the requirements, and can probably write an excellent set of test procedures, with a reasonable amount of time to pass QA.

Each time a system changes, a manual tester (maybe not the same manual tester) must dig out the same test procedures and try, as faithfully as humanly possible, to replicate all of those tests. If they are the original tester, that's probably going to be a reasonable amount of time -- but it's a violation of the DRY Principle (Don't Repeat Yourself)

(While DRY Principle refers to not replicating knowledge and replacing it with abstractions, an automated test could be considered an abstraction of a manual tester's process)

The cost of manually testing software

The problem of manual testing gets worse with time:

• The violation of DRY leads to paying for all the time needed for manual testers to run the exact same set of time-consuming and error-prone manual tests over, and over, and over; for many months and years into the future.
• If there's a rush to get a bugfix out the door, the tester doesn't have time to run all the tests, they'll just pick and choose a few at random and ignore 90% of the tests; that's a good way to let regression bugs into production.
• If the manual tester isn't the original QA tester who wrote the plan, they need to try to acquire all the knowledge which the original QA tester had in their head to try to execute the plan faithfully and take all the same considerations/observations into account
• The original tester perhaps assumed a bunch of things that future testers wouldn't know so future testers might interpret things differently because they're only human, and that could cause a lot of confusion, maybe having to go back to the business to question old requirements which haven't changed in years, possibly adding a lot of disruption/delay while the business tries to answer the question.
• The system itself grows increasingly complex and manual tests need to be maintained and updated just as automated tests need to be maintained and updated; but the additional time to keep re-running and maintaining manual tests also gets worse because manual test plans tend to be written in a very tedious, repetitive way.

When software is not 'soft'

Check this question on SE.SE for an excellent example of what happens when software has grown so unwieldy that it becomes extremely hard for a developer to work with, and causes them to ask a question like this: I've inherited 200K lines of spaghetti code -- what now?

Consider the origin of this code; it probably didn't start out as 200k lines of spaghetti, and it probably worked perfectly for its intended use-case at the time (in fact, the original programmers may have even thought to themselves that it was just a one-off bit of code that would never change).

Code entropy is often the natural and unavoidable fate for nearly all legacy code which doesn't have strong automated test coverage; developers may often not even think their code is going to last very long before someone scraps it but are usually proven wrong.

Software development generally happens on a backdrop of real-world human issues such as time/budget constraints, changing requirements, new/inexperienced developers, mistakes in manual testing, and developers having no "rails" to guide them when they break something by accident.

The best way any software delivery team can provide great value-for-money to their stakeholders is to focus on building very high-quality code which tests itself and therefore protects itself from code-entropy and future regression issues.

Even if the original developers delivered bug-ridden code as part of MVP and some of the tests were bad or wrong, automated test coverage establishes a baseline for quality (just as is the case for static code analysis tools), so it also assures that quality can only get better even if the baseline wasn't so great to start with; the fact that it can only go in a positive direction means avoiding code-entropy and gives the code a maintainable future.

Answer 7

• You're testing a function with another function ... so why did you stop there and test the function you just created?

You should test both. But you don't need to write a third test case for that, you test both by running them together. When the code is all in place you expect to see the 'green' success result. If you write the test first you can check it fails as you expect by running it before you write the production code. If you write the production code first you can do a quick edit to it (or to the test) and run it to check it fails in the way you expect.

It may be enough to do checks like that manually as you're coding, but if you want you can automate them via a mutation testing tool, that will automatically make changes to your code and check that that causes your tests to fail, which proves that they are really checking what your code is like.

• It's code, it has algorithm so how to be sure it is correctly written ?

You can't be completely sure, but you can be a lot surer than you could that the production code is correct if you don't test it. If you don't test it all you'd have very low confidence that it works. You can test it manually, which may give you good confidence if you do enough tests, but it can be tedious and you're unlikely to want to repeat every check every time you make a small change to the code.

The algorithm in your example should be something like:

result = add(3,5);
assertEquals(8, result);

This is very simple and much easier to read and understand than your own implementation of add would be. (Of course you wouldn't really create your own add function for integers since it's built into CPUs, but the same principle applies for more realistic examples - maybe you want to write a function that can add lengths. CPUs don't have this built in, but the test should be easier to read than the implementation:

result = length.of('12cm').add(length.of('5m'));
assertEquals(length.of('512cm'), result);

• And why stop with the expected output ? Why not testing that the input are of the correct type, because if I give characters to my add function it will fail, no ?

When you write a test you want to be clear about what you're testing and what you're not testing. When you want to write and test a function the inputs are not created by that function you want to make sure that the function behaves as you intend it to. The function doesn't create it's own inputs so you can't check what they are as part of testing that particular function. But you can and should consider whether it's possible that someone could pass inputs of the wrong type, decide how your function should respond to that, and consider writing a test to check it does so.

If you want to test that that the inputs to your function are right you need to think about where those inputs are created. If there's another part of your code that creates inputs and calls your function you can write test for that. This could be in addition to the test for the function, or you could decide to test thing that calls it instead, and let the function you started with be tested indirectly.

Answer 8

he will be sure that in final his functions are bullet proof

This is mostly where you're doing it wrong. Tests do not "ensure" that your functions are bullet-proof, and if you're telling your client that then you are digging yourself a very big hole. Tests add value only if the tests are of high quality.

I didn't think about this use case

And here we see the problem. You now had two chances to get this right, either when writing the code itself, or when designing your test cases. Tests do not remove the requirement on a developer to think.

Answer 9

Testing is needed because no one can fully understand the exact role of any -non-trivial sized- software.

Take as an example a C compiler like GCC or Clang. Understanding fully the C language (e.g. as specified in n1570) is non-trivial. Even with decades of C or C++ programming experience I cannot claim to know it fully. And many software (including GCC) have some parts of their C or C++ code which are machine generated (for example RefPerSys or Jacques Pitrat's CAIA system, described by an entire book: Artificial Beings, the conscience of a conscious machine).

Programs do fail.

In some cases, you can use a formal approach, e.g. with tools like Frama-C. Then you have to specify in some formal manner the behavior of your C program.

If that C (or C++) program is driving a neurosurgery robot or a robot on Mars, specifying it is a very difficult task. If it is just counting the number of vowels in some ASCII file, it is simpler (and bugs have less dramatic consequences).

Formal methods approaches reduce the number of bugs (or move them at a more abstract level). They don't remove all of them.

Read about Ariane flight v88. A rocket crash because of a software bug.

Answer 10

Many things can be said about testing, both the how and the why, but I want to strike at the heart of your issue:

It is unreasonable to expect that your tests catch every possible bug straight from the start.

You shouldn't hold yourself to that standard, because it's impossible to meet. You also shouldn't avoid testing because you can't meet that standard.

Test suites strengthen over time. You write a test suite which covers all the cases you could think of. Let's for now assume that you totally forgot about division by zero.

A bug occurs in production code. You retrace the steps and figure out that it stems from trying to divide by zero. You think to yourself "Huh, why did the test suite not catch that?"

You notice the test suite doesn't test for division by zero, and therefore you now add this test to the suite. The test suite will from now on test for division by zero.

You can't expect to catch every bug straight from the start, but you should expand your test suite whenever a new bug appears that wasn't already being caught by the test suite.

---

In conclusion:
Over time, your test suite will become stronger and stronger. Also, over time you will become more experienced in writing test suites, which means that you're likely going to write a more comprehensive test suite from the beginning. But the odds are that there's always some fringe bug that you forgot to write a test for, thus having to add it into the test suite at a later point (when you realize that this test has been missing all along).

---

Special mention:

"So I pay you more because you said that the tests you've written will ensure a bug free app... but it's not ? And now I have to pay you to debug it ?"

Tests do not remove the need to troubleshoot or debug. Tests lower the average effort required to find the source of a bug by confirming which parts do/don't work, thereby helping you zero in on the location of the bug in question.

Sometimes, tests catch things that then never end up being a bug in production, but this is not the only reason to have tests. Sometimes, bugs still make it into production but your tests help you eliminate potential places where the bug might be situated (which drastically helps reduce the amount of time spent looking for the source of the bug).

Without tests, you'd just be manually testing what the automated tests would otherwise be doing for you. And the automated tests are much better and faster than a developer who is only human and prone to error.

Answer 11

First let me say I am no testing guru, and not only do I not do TDD, I don’t write as many test as I think I should.

That said, you really need to take a step back and ask “What is unit testing”, and the answer to that is “a continuous and automatic validation that the code under test performs as expected in a given circumstances”. IOW it should resolve some uncertainty about whether the code handles a particular scenario.

The test code doesn’t need a unit test because the act of testing it IS the unit test for it —- if the test fails then either your unit test is wrong or your expectations were wrong. It’s impossible to say which is wrong without reference to an 3rd source of truth (it may very well be both are wrong).

A unit test is a way to verify that a piece of code does what it is expected to. A simple add function such as you describe can test 5+3, but probably doesn’t need to, what it should be testing is something like int.max + 1 or int.min + -1.

In general unit tests should tests business rules, specifications and algorithms. To put it another way, a unit test should increase your confidence that under a specific scenario you know what will happen.

In short, what you are testing should have some complexity, either in the code or in the data. Often the code that needs to be tested can’t because it is entwined with code that is irrelevant. Making it testable can be beneficial, even if it isn’t actually tested, because it separates the functions into separate code blocks, making the code easier to understand.