In computing, software engineering, and software testing, a test oracle is a mechanism for determining whether a test has passed or failed. The use of oracles involves comparing the output of the system under test, for a given test-case input, to the output that the oracle determines that product should have. The term "test oracle" was first introduced in a paper by William E. Howden. Additional work on different kinds of oracles was explored by Elaine Weyuker. Oracles often operate separately from the system under test. However, method postconditions are part of the system under test, as automated oracles in design by contract models. Determining the correct output for a given input is known as the oracle problem or test oracle problem, which is a much harder problem than it seems, and involves working with problems related to controllability and observability. Various methods have been proposed to alleviate the test oracle problem. A popular technique is metamorphic testing.
Categories
A research literature survey covering 1978 to 2012 found several potential categorisations for test oracles.
A derived test oracle differentiates correct and incorrect behaviour by using information derived from artefacts of the system. These may include documentation, system execution results and characteristics of versions of the system under test. Regression test suites are an example of a derived test oracle - they are built on the assumption that the result from a previous system version can be used as aid for a future system version. Previously measured performance characteristics may be used as an oracle for future system versions, for example, to trigger a question about observed potential performance degradation. Textual documentation from previous system versions may be used as a basis to guide expectations in future system versions. A pseudo-oracle falls into the category of derived test oracle. A pseudo-oracle, as defined by Weyuker, is a separately written program which can take the same input as the program/system under test so that their outputs may be compared to understand if there might be a problem to investigate.
Implicit
An implicit test oracle relies on implied information and assumptions. For example, there may be some implied conclusion from a program crash, i.e. unwanted behaviour - an oracle to determine that there may be a problem. There are a number of ways to search and test for unwanted behaviour, whether some call it negative testing, where there are specialized subsets such as fuzzing. There are limitations in implicit test oracles - as they rely on implied conclusions and assumptions. For example, a program/process crash may not be a priority issue if the system is a fault-tolerant system and so operating under a form of self-healing/self-management. Implicit test oracles may be susceptible to false positives due to environment dependencies.
Human
When specified, derived or implicit test oracles cannot be used, then human input to determine the test oracles is required. These can be thought of as quantitative and qualitative approaches.
A quantitative approach aims to find the right amount of information to gather on a system under test for a stakeholder to be able to make decisions on fit-for-purpose / release of the software.
A qualitative approach aims to find the representativeness and suitability of the input test data and context of the output from the system under test. An example is using realistic and representative test data and making sense of the results.
These can be guided by heuristic approaches, i.e. gut instinct, rule of thumb, checklist aids and experience to help tailor the specific combination selected for the program/system under test.
Examples
Common oracles include:
specifications and documentation. A formal specification used as input to model-based design and model-based testing would be an example of a specified test oracle. Documentation that was not a formal specification of the product would typically be a derived test oracle, e.g. a usage or installation guide, or a record of performance characteristics or minimum machine requirements for the software.
other products. This is an example of a derived test oracle, a pseudo-oracle.
a heuristic oracle that provides approximate results or exact results for a set of a few test inputs
a statistical oracle that uses statistical characteristics, for example with image analysis where a range of certainty/uncertainty is defined for the test oracle to pronounce a match or not. This would be an example of a human test oracle.
a consistency oracle that compares the results of one test execution to another for similarity. This is an example of a derived test oracle.
a model-based oracle that uses the same model to generate and verify system behavior, an example of a specified test oracle.
