What is the most complete way to test an imperative function?

Question

What is the most complete way to test an imperative function? By "complete" I mean, something that is robust, that tests all the edge cases of the function, that so much as "proves" it to be correct. Ideally it isn't brute force, but if it has to be that's fine.

Say you have a really complicated function. You break it apart into 10 layers of simpler and simpler functions, down to 2 or 3 line functions. What is the ideal ideal in terms of what you how you can test your large function? Not in the sense of if we have time constraints. Don't worry about it taking a long time to write the tests. I am wondering in theory (so it can be applied in practice without time constraints) what the best way to test is (by best I mean most completely/thoroughly).

They don't necessarily have to be tests, they could be things like TLA+ specifications, or symbolic executions or model checking sorts of things. I am just trying to capture into one post what is the ideal for testing an imperative function like in JavaScript, if you had a ton of time and wanted to be extremely thorough.

I would think doing model checking on your function would be enough, but maybe it is not enough, I don't know. Proving functions in the real-world correct seems to be a lost cause, from the answers I've received, especially async or side-effect-ridden or concurrency functions. Writing unit or integration tests isn't specific enough... What should the unit or integration tests include generally, to be complete coverage of every edge case? Is traditional unit/integration TDD enough, or do you need more?

Answer 1

This answer is a frame challenge, because I feel like you're trying to have your cake and eat it too.

There are two ways people write tests in the real world: reasonably, and pedantically. These are just names I picked, they are not meant to imply one is better than the other, but they are very different beasts.

Reasonable testing strategies rely on only writing tests for reasonably expected situations, and writing tests ad hoc for any additional situations that are concretely encountered. The main goal here is to prevent time wastage due to pedantically testing every possible situation, even ones that are theoretically possible but never actually happen.
The pro is that you keep your testing effort down to what's relevant, the con is that you're more likely to find a bug, realize that a test could've uncovered this sooner, and now write a test for it.

Pedantic testing strategies, on the other hand, prefer that time wastage in writing every possible test, because they really want to avoid having to iteratively improve their testing suite. This is much preferred in cases where software can't easily be patched (e.g. embedded systems) or where mistakes can cost lives or serious bodily harm (e.g. medical equipment, safety systems, ...).
The pro is that you know your code inside out, the con is that is takes multiple times longer to test your software, and your codebase (and indirectly your dev team spirit) becomes very change-unfriendly due to the impact such a change may have on your test suite (and the effort that this entails).

By "complete" I mean, something that is robust, that tests all the edge cases of the function, that so much as "proves" it to be correct.

Okay, so this puts you squarely in the pedantic testing strategy approach.

• It should also be sophisticated, and not just manual
• I also don't think it's ideal to say write one test for every possible combination of something (given we have a lot of time), that is not sophisticated enough. There should be a generic solution or automatic solution in that case.

This is why I say you're trying to have your cake and eat it. You're asking for a test suite that carries all the confidence of having done the complete legwork, without having done all of the actual leg work.

It is also impossible to at the same time test/confirm every edge case, while not testing every input. The simple example is as follows:

public string MySpecialMethod(int input)

I need you to test this method. But, by your own standard, you should both make sure that there is no unexpected behavior for any int value that is passed into the method, and additionally you are not allowed to test every input value.
Big spoiler: I wrote some special if (input == ...) throw new Exception(); logic, and you don't know what the ... is. Well, maybe you'll get lucky and happen to encounter the value in the limited subset of possible input values that you're going to test.

But can you really be confident if your test requires a wishful hope that you happened to encounter this issue? No. It's the opposite of confidence, or having tested all edge cases.

You may argue that in a white box testing scenario, you're allowed to see inside the logic, but you're missing the point. I used a blatant if throw example here, but bugs are generally not blatant.
Since you are asking for complete confidence in the test output, my argument applies to any possible bug that could arise due to any number of possibilities, which can include certain combinations of input values, which in turn suggests that you can't have perfect confidence without having tried those combinations of input values.

What should the unit or integration tests include generally, to be complete coverage of every edge case?

This is simply not meaningfully answerable without so much as a faint grasp on what the object under test is.

If you want a general blanket answer that covers every base and gives you perfect confidence, without any concrete example, the only answer you're going to get is rigorous brute force.

---

To conclude, I don't know what you're expecting from having posted this question. If there was an answer here that is simultaneously easy to explain, gives you perfect confidence, is sophisticated, covers every base for every possible object under test, and doesn't cost ungodly amounts of time to write/run these tests; it would literally be the panacea to testing strategies and we would all be doing this and nothing else any more.

Answer 2

There are many measures of test coverage... each requiring more test cases to achieve.

• Statement Coverage shows that you have tested every statement (at least once)

• Branch Coverage shows that you have tested every branch (imagine you've drawn a flowchart)

• Decision Coverage shows that you have tested every decision (ie each true/false exit of a decision box on your flowchart) - this is very closely related to Branch Coverage

• Condition Coverage shows that you have tested each Condition (the sub-parts of a Decision) at least once

• Condition/Decision Coverage shows that you have tested each Condition and each Decision

• Modified Condition/Decision Coverage (MC/DC) shows that (in addition) you've shown each Condition independently

Over and above these, you need to bounds check all inputs, to demonstrate no overflow or underflow (both at the inputs and at intermediate stages).

Just testing edge cases is not enough

--

In the aerospace world, for the highest Design Assurance Levels, MC/DC is required...