2

I am currently checking through the syllabus for the ISTQB "Technical Test Analyst" certification. This syllabus (henceforth called "TTA syllabus") contains a chapter dedicated to "condition testing" (apparently this is also called "condition coverage" or "condition coverage testing".)

Condition testing as defined in the TTA syllabus strikes me as an exceedingly strange and possibly anachronistic thing that has more to do with Grandma's Ham Story than with software testing. Let me explain...

What is "condition testing" according to the TTA syllabus?

Referring to the "Advanced Level Syllabus - Technical Test Analyst, Version 2012" of the ISTQB, available here, "condition testing" is defined as follows on page 12:

Compared to decision (branch) testing, which considers the entire decision as a whole and evaluates the TRUE and FALSE outcomes in separate test cases, condition testing considers how a decision is made. Each decision predicate is made up of one or more simple “atomic” conditions, each of which evaluates to a discrete Boolean value. These are logically combined to determine the final outcome of the decision. Each atomic condition must be evaluated both ways by the test cases to achieve this level of coverage.

Applicability

Condition testing is probably interesting only in the abstract because of the difficulties noted below. Understanding it, however, is necessary to achieving greater levels of coverage that build upon it.

So let's clarify the words

Suppose we have some software-under-test on which we want to do whitebox testing (i.e. we have the source code). The code naturally contains many of those decision predicates that we all know and love and that show up after strings like if, while, until etc.

An example of the decision predicate might be:

((x>y+z) AND (y<-3)) OR ((z²+x²<4) AND (z≤y))

An example of a decision predicate

This decision predicate accepts a triple (x,y,z) of integer values (say) and outputs a boolean, b_out.

The first-level functions which map (x,y,z) to {TRUE,FALSE} are called conditions (also atomic conditions) in the TTA syllabus.

Condition testing coverage

One attains "condition testing coverage" by running test cases until all the conditions found in the decision yielded at least once true and at least once false.

One can thus achieve condition testing coverage by running the following five test cases (for example):

Truth table achieving condition testing coverage

Each of the the b0, b1, b2, b3 shows up at least once with true and at least once with false. In some test cases, we do not care about a given value as it has no influence on the condition whose output we want to get right.

Addendum: Decision coverage

Incidentally, the truth table shows also that "decision coverage" has been attained for this decision, as b_out shows up at least once with true and at least once with false, so both branches of any code would be covered.

And now the question

What is "condition testing" as defined above actually good for?

It won't help you ascertain the correctness of the decision predicate at all.

That is best done by a code inspection, letting a second team write the same expression and comparing outputs or running test cases at "boundary values" (e.g. here, for y = -4, -3, -2).

Checking the individual conditions strikes me to be more about testing the CPU's ALU or maybe testing the compiler output. It is certainly suitable for checking manually written assembler code or verifying manually assembled logic.

And this makes me think that "condition testing" could well be a leftover from the days where conditions were actually wired up on panels (for example in the ENIAC) and might thus be subject to wiring bugs. These days, the condition is written in high-level code and is exactly the one you conceptually want. While review might be useful, a testing of the condition is just a waste of time.

Or am I missing something?

Addendum: Literature Search

A search of the IEEE Xplore library for "condition testing" yields only two papers relevant to software (all the others seem relevant to hardware only), both by K.C. Tai of the Department of Computer Science of North Carolina State University.

Checking one of those, Condition-based software testing strategies reveals that the author uses the term condition in the sense of decision above, i.e. in this paper "condition testing" is actually "decision testing". The condition as used in the TTA syllabus is called simple condition. It seems the TTA syllabus definitions are not widely used.

From the abstract:

A computer program consist of statements, such as IF and WHILE statements, that contain conditions, which are combinations of Boolean and relational expressions. A testing approach, referred to as condition testing, is to test a program by focusing on testing the conditions in this program. A number of condition testing strategies have been developed, but they are not effective for detecting errors in complicated conditions. In this paper, we define two condition testing strategies, based on the detection of Boolean and relational expression [i.e. expression of the form E1 op E2, where E1 and E2 are arithmetic expressions and op is one of six possible relational operators: < <=, =, !=, >, >=] errors in a condition. For these two condition testing strategies, we show some theoretical properties and explain why they are practical and effective for testing programs containing complicated conditions.

  • This is NOT a duplicate of "Is test coverage an adequate measure of code quality?". I really would like to know how anyone can come to that conclusion. This one is about "condition testing", for starters. "Condition testing" is a specific case of coverage. Maybe we should pack up Mathoverflow because it's all about "functions" anyway? – David Tonhofer Apr 28 '14 at 13:40
  • I don't know where you got the idea that condition coverage (or any other code coverage metric) has anything to do with the "correctness" of the code. It has nothing to do with it. Such metrics are a way to measure the completeness of the test suite, nothing more. And some coverage metrics are better at that than others. In the case of condition coverage, it's a better/more powerful metric than branch/decision coverage or statement coverage. That is the reason why condition coverage is worth knowing about. – Rogério Jan 1 at 21:55
  • @Rogério It's been some time but probably because it is called "condition testing" not "subcondition coverage testing". It's still a pointless exercise because why would anyone want to cover the N-dimensional space of the condition's input variables? Unless it's done by the the person designing the condition - in which case he/she will take the code expressing the condition and test in separation using few test cases, but not in the final product code. One would hope... – David Tonhofer Jan 2 at 10:55
  • From the first quote in the question (which mentions "atomic conditions"), we see that a "condition" is what you refer as a "subcondition". That is, in the if (a && b) decision, we have two conditions, a and b which get separately evaluated (to true/false each, with b only getting evaluated when a is true) in order to produce the true/false outcome of the decision/branch. So, for full decision/branch coverage, two tests are sufficient, while full condition coverage takes three tests, and these three tests also cover the true and false decision outcomes (ie, both branches). – Rogério Jan 4 at 21:06
3

Condition testing is a direct consequence of the fact that each if in your software creates a "bifurcation" (essentially breaking your code up into two separate bits of code), thereby increasing the overall cyclomatic complexity of your code.

In order to get full code coverage in a white box testing scenario, you need to have some knowledge of the if conditions, so that you can make sure that each code branch is being exercised during testing.

As to the "correctness" of the code, that can be ascertained by writing meaningful unit tests. That is, for each input that exercises a particular path in the code, you can expect some output, the result of which demonstrates that the correctness that you are after has been achieved.

By "meaningful," I mean that, generally your unit tests will demonstrate that some contract or requirement has been fulfilled. But you still need to make sure you exercise all code paths, so that there are no unpleasant surprises later.

  • But that is what "decision testing" is for and what "decision coverage" is about. You get true or false on the decision, so go down the two branches (still not a useful metric, but that is another discussion) – David Tonhofer Apr 15 '14 at 18:12
  • Then what is your question about? See my ninja edit while you were writing your comment. – Robert Harvey Apr 15 '14 at 18:13
  • Your Ninja edit is correct. The question boils down to: Why mention Condition Testing at all in a syllabus mention for a professional certification? It seems either inadequate or appropriate to other problems. Decision testing has its own chapter in the syllabus -- that is what decision coverage is about -- and does indeed make sense (less so today than in the days of COBOL and FORTRAN. Now exceptions may occur and whole functions can be passed around, attached to objects at runtime, or even constructed on the fly). Should I write ISTQB to just remove the baggage of condition testing :-) – David Tonhofer Apr 15 '14 at 19:14
  • 1
    Baggage? Apparently I haven't stated my case very well. I have bolded the most important paragraph in my answer. – Robert Harvey Apr 15 '14 at 19:15
  • No -- unit tests do not even enter the picture, nor do contracts or requirements (don't worry, I know about these). You can do condition tests only by ripping out the decision, then test its conditions. But that doesn't tell you anything about anything. Except if you are testing the hardware implementing the condition! – David Tonhofer Apr 15 '14 at 19:19
-1

The best answer I could find (and which also applies to statement coverage testing and other types of coverage testing) is given by this page from atollic.com "Why code coverage analysis?"

More technically, code coverage analysis finds areas in your program that is not covered by your test cases, enabling you to create additional tests that cover otherwise untested parts of your program. It is thus important to understand that code coverage helps you understand the quality of your test procedures, not the quality of the code itself.

And this seems to be quite relevant here. If you have a test case set that manages to set the individual conditions of any decision to TRUE or FALSE, then you are quite likely invoking the code under test with a rather exhaustive set of input values!

Good condition coverage does not tell you much about the code under test (unless the code blows up or generates detectable faults) but gives you confidence in your test case set.

Addendum:

This is interpretation is correct. In the paper The effect of program and model structure on MC/DC test adequacy coverage (Ajitha Rajan and Mats Heimdahl, University of Minnesota, ICSE, 2008), which is about the related "Modified Condition/Decision Coverage" (MC/DC), one reads:

Adequacy of test suites is currently measured using the MC/DC metric on source code (or on a model in model-based development) ... Test adequacy metrics defined over the structure of a program, such as statement coverage, branch coverage, and decision coverage, have been used for decades to assess the adequacy of test suites. Such criteria can be useful tools when evaluating a testing effort.

The paper also concludes that MC/DC coverage is highly sensitive to the structure of the SUT. Purely syntactic rearrangements can change MC/DC coverage dramatically. This is of course because MC/DC just works on the syntax, not on the data flow, and is therefore a rather suspicious measure in any case. But that is another problem entirely.

  • 1
    this seems to merely repeat what was stated in another answer in a duplicate question – gnat Apr 23 '14 at 17:23
  • @gnat Yes, so what of it? It occurred to me that it actually applies to all coverage testing so I added it over there too. Amazingly, I already got two downvotes with no comment as to why. This seems to point to a particular sad state of the testing community. "Just ceremony but no understanding", I reckon. All these coverage measures strike me as borderline useless. Expect for finding dead code and for that static analysis is best anyway. – David Tonhofer Apr 23 '14 at 21:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.