-4

What is the most complete way to test an imperative function? By "complete" I mean, something that is robust, that tests all the edge cases of the function, that so much as "proves" it to be correct. Ideally it isn't brute force, but if it has to be that's fine.

Say you have a really complicated function. You break it apart into 10 layers of simpler and simpler functions, down to 2 or 3 line functions. What is the ideal ideal in terms of what you how you can test your large function? Not in the sense of if we have time constraints. Don't worry about it taking a long time to write the tests. I am wondering in theory (so it can be applied in practice without time constraints) what the best way to test is (by best I mean most completely/thoroughly).

They don't necessarily have to be tests, they could be things like TLA+ specifications, or symbolic executions or model checking sorts of things. I am just trying to capture into one post what is the ideal for testing an imperative function like in JavaScript, if you had a ton of time and wanted to be extremely thorough.

I would think doing model checking on your function would be enough, but maybe it is not enough, I don't know. Proving functions in the real-world correct seems to be a lost cause, from the answers I've received, especially async or side-effect-ridden or concurrency functions. Writing unit or integration tests isn't specific enough... What should the unit or integration tests include generally, to be complete coverage of every edge case? Is traditional unit/integration TDD enough, or do you need more?

6
  • why does it matter that it's imperative? – user253751 Feb 23 at 9:27
  • 3
    This question is at the same time too broad sweeping and inherently dismissive of what things shouldn't be. This really needs to be whittled down into a concrete question with a concrete example. Asking how to test something completely without having an idea of what that thing is or might be is am impossible task which would require a college course amount of explanation. Also, "there should be X" isn't productive, as it's counterintuitive to both ask a question that's so open-ended without concretions, and on the other hand already dismiss several reasonable answers with no explanation. – Flater Feb 23 at 9:28
  • I thought a concrete testing example would be better for stackoverflow. I thought this site was about more workflow, methodologies, and testing, among other closely related points... This is a specific question, how to robustly test a function. This is a reasonable and valid question for this site if you think about it. Thanks for the explanation. I don't mean to be dismissive, I am stating my experience. – Lance Pollard Feb 23 at 9:28
  • 2
    "By "complete" I mean, something that is robust, that tests all the edge cases of the function, that so much as "proves" it to be correct. It should also be sophisticated, and not just manual brute force." Impossible Here, I wrote a ForLance(int) method for you. I need you to test it and make sure that it doesn't throw any exceptions for any int value you pass into it. Oh, and by your own ideas, you're not allowed to test every int value. It's impossible for you as a test writer to know what arbitrary int value I may have picked to throw an exception. You'd need to go looking for it. – Flater Feb 23 at 9:31
  • You've edited the question, but you've only removed the clear markers (which have been pointed out) that showcase the core issue with the question, without changing the question itself. The issue I tried to point out wasn't related to that one sentence, it's the underlying premise of the question itself. – Flater Feb 23 at 10:02
3

This answer is a frame challenge, because I feel like you're trying to have your cake and eat it too.

There are two ways people write tests in the real world: reasonably, and pedantically. These are just names I picked, they are not meant to imply one is better than the other, but they are very different beasts.

Reasonable testing strategies rely on only writing tests for reasonably expected situations, and writing tests ad hoc for any additional situations that are concretely encountered. The main goal here is to prevent time wastage due to pedantically testing every possible situation, even ones that are theoretically possible but never actually happen.
The pro is that you keep your testing effort down to what's relevant, the con is that you're more likely to find a bug, realize that a test could've uncovered this sooner, and now write a test for it.

Pedantic testing strategies, on the other hand, prefer that time wastage in writing every possible test, because they really want to avoid having to iteratively improve their testing suite. This is much preferred in cases where software can't easily be patched (e.g. embedded systems) or where mistakes can cost lives or serious bodily harm (e.g. medical equipment, safety systems, ...).
The pro is that you know your code inside out, the con is that is takes multiple times longer to test your software, and your codebase (and indirectly your dev team spirit) becomes very change-unfriendly due to the impact such a change may have on your test suite (and the effort that this entails).

By "complete" I mean, something that is robust, that tests all the edge cases of the function, that so much as "proves" it to be correct.

Okay, so this puts you squarely in the pedantic testing strategy approach.

  • It should also be sophisticated, and not just manual
  • I also don't think it's ideal to say write one test for every possible combination of something (given we have a lot of time), that is not sophisticated enough. There should be a generic solution or automatic solution in that case.

This is why I say you're trying to have your cake and eat it. You're asking for a test suite that carries all the confidence of having done the complete legwork, without having done all of the actual leg work.

It is also impossible to at the same time test/confirm every edge case, while not testing every input. The simple example is as follows:

public string MySpecialMethod(int input)

I need you to test this method. But, by your own standard, you should both make sure that there is no unexpected behavior for any int value that is passed into the method, and additionally you are not allowed to test every input value.
Big spoiler: I wrote some special if (input == ...) throw new Exception(); logic, and you don't know what the ... is. Well, maybe you'll get lucky and happen to encounter the value in the limited subset of possible input values that you're going to test.

But can you really be confident if your test requires a wishful hope that you happened to encounter this issue? No. It's the opposite of confidence, or having tested all edge cases.

You may argue that in a white box testing scenario, you're allowed to see inside the logic, but you're missing the point. I used a blatant if throw example here, but bugs are generally not blatant.
Since you are asking for complete confidence in the test output, my argument applies to any possible bug that could arise due to any number of possibilities, which can include certain combinations of input values, which in turn suggests that you can't have perfect confidence without having tried those combinations of input values.

What should the unit or integration tests include generally, to be complete coverage of every edge case?

This is simply not meaningfully answerable without so much as a faint grasp on what the object under test is.

If you want a general blanket answer that covers every base and gives you perfect confidence, without any concrete example, the only answer you're going to get is rigorous brute force.


To conclude, I don't know what you're expecting from having posted this question. If there was an answer here that is simultaneously easy to explain, gives you perfect confidence, is sophisticated, covers every base for every possible object under test, and doesn't cost ungodly amounts of time to write/run these tests; it would literally be the panacea to testing strategies and we would all be doing this and nothing else any more.

11
  • Even in safety-critical embedded software, 'reasonable testing strategies' are used, as even there you don't have the time to cover the entire state-space of the device. What is considered reasonable might be a larger/deeper test than for other kinds of software though. – Bart van Ingen Schenau Feb 23 at 10:55
  • @BartvanIngenSchenau For sure. But I happen to have experience with writing a test suite for an automated large infrastructure safety system and they did in fact test every permutation of anything that was attached to a "critical output". The example they gave to explain is that the direction a fan blows in is a single input bit, but it makes all the difference between letting the fire department safely enter a burning tunnel or aiming a 10m wide flamethrower cannon at them. – Flater Feb 23 at 11:12
  • Surely it is the theoretically possible but never actually happen ones you should be testing the behaviour of, because Murphy has been proven to be an optimist - and they will most definitely happen in the real world! You know, the line of code that has the comment can never reach here that gets executed with alarming regularity! – Andrew Feb 23 at 11:40
  • That's what I'm looking for, exactly! The panacea haha. What is the panacea as it stands today? Formal proofs seemed like it, until I got a lot of negative responses when posing questions about async and side effects. – Lance Pollard Feb 23 at 11:49
  • @Andrew: Keep in mind that defining behavior means enshrining it into an analysis, which means developers needs to talk to analysts to know if Y is the correct behavior in response to situation X. This can lead to inane discussions and pointless meetings, which is a needless drain on resources if the situation under discussion doesn't even happen. When you bought your shoes, did you vet them on how they behave in -25°C temperatures? Or did you not bother because you don't live in a region that hits these kinds of temperatures? It's a silly example but the point stands. – Flater Feb 23 at 11:50
1

There are many measures of test coverage... each requiring more test cases to achieve.

  • Statement Coverage shows that you have tested every statement (at least once)

  • Branch Coverage shows that you have tested every branch (imagine you've drawn a flowchart)

  • Decision Coverage shows that you have tested every decision (ie each true/false exit of a decision box on your flowchart) - this is very closely related to Branch Coverage

  • Condition Coverage shows that you have tested each Condition (the sub-parts of a Decision) at least once

  • Condition/Decision Coverage shows that you have tested each Condition and each Decision

  • Modified Condition/Decision Coverage (MC/DC) shows that (in addition) you've shown each Condition independently

Over and above these, you need to bounds check all inputs, to demonstrate no overflow or underflow (both at the inputs and at intermediate stages).

Just testing edge cases is not enough

--

In the aerospace world, for the highest Design Assurance Levels, MC/DC is required...

6
  • Would this make a function completely tested, you need nothing else? – Lance Pollard Feb 23 at 9:36
  • 1
    Well... I suggest nothing else is quite a dismissive way of noting testing all paths, all decisions, all conditions independently, all bounds, all ranges - and that the correct results are produced... but yey. – Andrew Feb 23 at 10:06
  • @LancePollard, that would make a pure function completely tested. As soon as state is involved, you can get a whole battery of additional tests, like N-state transition tests of a state machine. – Bart van Ingen Schenau Feb 23 at 11:00
  • @BartvanIngenSchenau - I thought I'd covered those with the coverages, if the states are relevant to the function? Of course, in a state machine, you need to test each state and each transition... – Andrew Feb 23 at 11:36
  • 1
    @Andrew, that would depend if you consider the state of the system/universe an input to the function or not. I typically don't. – Bart van Ingen Schenau Feb 23 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.