1

I know the definition of an testing oracle but to me, it seems to be overrated. I have a specification, why should I bother with thinking about oracles? I just compare what the spec wants and what the program does. Or I have the use cases, whatever. When I read about oracles in books, it looks as if it a science on its own. Is there anything more behind this concept? Maybe I just did not understand its point.

An oracle consists of a generator and a comparator. What is the meaning of the comparator if you just check whether an actual output equals to an expected output? How else could it be checked?

4
  • 4
    What are you defining an "oracle" as? You say "I have a specification, why should I bother with thinking about oracles?", but in the Wikipedia article Oracle (software testing) the very first example given of an oracle is "specifications and documentation". Apr 14 '13 at 10:58
  • 1
    Well Wikipedia is not an authoritative source IMO. Books say "it is a mechanism to determine whether the test has passed or not". or "In software testing, the mechanism used to generate expected results is called an oracle."
    – John V
    Apr 14 '13 at 11:11
  • @Carson63000 Could you please then explain the concept of "Comparator" and "Generator" to me as well? Because I do not think it is clear from saying "Oracle is a specification".
    – John V
    Apr 14 '13 at 11:19
  • 1
    @user970696 Wikipedia is usually a wonderful source for programming topics. There are a lot of books out there with bad advice that never gets changed, but WP articles are community projects and garbage tends to get cleaned out. Apr 14 '13 at 13:56
2

I have a specification, why should I bother with thinking about oracles? I just compare what the spec wants and what the program does.

The thing is: what if you don't have a specification, or the specification is more complex than just a list of inputs and expected outputs? Some examples:

  • You have a domain expert instead of a complete written specification
  • The specification says: the system must behave exactly like the legacy system except for these here listed changes
  • Some of the system's behavior is random, the specification lists constraints like statistical tests
  • You're implementing a complex numerical algorithm to solve differential equations. The output will contain numerical errors, in some cases more than in others

"Oracle" is simply a more general term than specification.

What is the meaning of the comparator if you just check whether an actual output equals to an expected output?

The last two points above are examples where the check is much more complex than a simple equality test.

1

I think its a nice way of saying Gospel truth. For some programs the exact output is difficult to get for different data sets (algorithms, computations, accountancy, taxes, insurance ...). The tester/ test suite needs the correct answer to compare actual software output. The specs will have the method & some samples but not the answer for current data set & state.

The oracle in this case is parallel software or person or method to get the answer. For you might be your neighbor, for a software suite might be the text file where you have entered the expected value.

Oracle is the one with the answers.

Example : When you have one data set, say for a tax application, a person earning X, with Y deductions, and other environmental factors like number of children, etc The comparator would take this data and come up with the tax due, then compare it what the program outputs; to see if it got it right. If they do not match, one of them has it wrong. Hopefully the app or the oracle is not a good oracle.

5
  • I am too interested in the "comparator" part the sources mention. What is meant by that?
    – John V
    Apr 14 '13 at 12:12
  • see updated answer
    – tgkprog
    Apr 14 '13 at 12:30
  • It sounds more like generator which generates the expected outcome. I think that comparison is just what we do, simple equality check? Then I do not know why it is mentioned as a part of an oracle.
    – John V
    Apr 14 '13 at 14:00
  • 1
    @user970696: What about cases where the comparison is less straightforward. For example when verifying that a random-number-generator is sufficiently random. Then the generator part of an Oracle is trivial (to non-existing) and the comparator has to do a lot of work (mostly statistical analysis). Apr 14 '13 at 17:22
  • @bart-van-ingen-schenau i think that is still the oracle. its just the way it verifies the output is different. the specs are special. same would go for a game ... random or a fair dividing of resources that are not all the same
    – tgkprog
    Apr 14 '13 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.