Specification by example


Specification by example is a collaborative approach to defining requirements and business-oriented functional tests for software products based on capturing and illustrating requirements using realistic examples instead of abstract statements. It is applied in the context of agile software development methods, in particular behavior-driven development. This approach is particularly successful for managing requirements and functional tests on large-scale projects of significant domain and organisational complexity.
Specification by example is also known as example-driven development, executable requirements, acceptance test–driven development, Agile Acceptance Testing, Test-Driven Requirements.

Advantages

Human brains are generally not that great at understanding abstractions or novel ideas/concepts when first exposed to them, but they’re really good at deriving abstractions or concepts if given enough concrete examples. The more examples we are given, the more likely we are to correctly understand the intended meaning. Also, by using concrete examples, they become more familiar and relatable to something similar to our past experiences, which generally makes them easier to understand.
Successful application of Specification by example is documented to significantly reduce feedback loops in software development, leading to less rework, higher product quality, faster turnaround time for software changes and better alignment of activities of various roles involved in software development such as testers, analysts and developers.

Examples as a single source of truth

A key aspect of specification by example is creating a single source of truth about required changes from all perspectives. When business analysts work on their own documents, software developers maintain their own documentation and testers maintain a separate set of functional tests, software delivery effectiveness is significantly reduced by the need to constantly coordinate and synchronise those different versions of truth. With short iterative cycles, such coordination is often required on weekly or biweekly basis. With Specification by example, different roles participate in creating a single source of truth that captures everyone's understanding. Examples are used to provide clarity and precision, so that the same information can be used both as a specification and a business-oriented functional test. Any additional information discovered during development or delivery, such as clarification of functional gaps, missing or incomplete requirements or additional tests, is added to this single source of truth. As there is only one source of truth about the functionality, there is no need for coordination, translation and interpretation of knowledge inside the delivery cycle.
When applied to required changes, a refined set of examples is effectively a specification and a business-oriented test for acceptance of software functionality. After the change is implemented, specification with examples becomes a document explaining existing functionality. As the validation of such documents is automated, when they are validated frequently, such documents are a reliable source of information on business functionality of underlying software. To distinguish between such documents and typical printed documentation, which quickly gets outdated, a complete set of specifications with examples is called Living Documentation.

Key practices

Teams that apply Specification by example successfully commonly apply the following process patterns:
Software teams that apply specification by example within a Scrum framework typically spend 5%-10% of their time in refining the product backlog, including specifying collaboratively, illustrating requirements using examples and refining examples.

Applicability

Specification by example applies to projects with sufficient organisational and domain complexity to cause problems in understanding or communicating requirements from a business domain perspective. It does not apply to purely technical problems or where the key complexity is not in understanding or communicating knowledge. There are documented usages of this approach in domains including investment banking, financial trading, insurance, airline ticket reservation, online gaming and price comparison. A similar approach is documented also in a nuclear-power plant simulation project.
Tests based on shared examples fit best in the category of tests designed to support a team while delivering software from a business perspective - ensuring that the right product is built. They do not replace tests that look at a software system from a purely technical perspective or tests that evaluate a product after it was developed.

History

The earliest documented usage of realistic examples as a single source of truth, requirements and automated tests, on software projects is the WyCash+ project, described by Ward Cunningham in the paper A Pattern Language of Competitive Development in 1996. The name Specification by Example was coined by Martin Fowler in 2004.
Specification by Example is an evolution of the Customer Test practice of Extreme Programming proposed around 1997 and Ubiquitous Language idea from Domain-driven design from 2004, using the idea of black-box tests as requirements described by Weinberg and Gause in 1989.

Derived works

Example Mapping

Example Mapping is a simple technique that can steer the conversation and derive Acceptance criteria within 30 minutes.The process involves breaking each stories into Rules and Examples and documented in the form of Specification by examples. Example Mapping was first introduced by Matt Wynne in the 2015 Agile alliances conference and is one of the wildly used techniques in the BDD world.

SHEQC grooming

Similar to "Example mapping" SHEQC grooming enables teams to groom a complex user story in less than 30 to 45 min using a concept called as continuous grooming using design thinking techniques. SHEQC uses Specification by example as the standard for documenting scenarios. The process involves the double diamond rule for brainstorming and the out come is a set of question and Acceptance criteria again documented in the form of Specification by example for the story. SHEQC grooming was first introduced in the Innovations in software engineering conference ISEC2019, WESSEE by Ranjith Tharayil and later published in XP2019 conference as one of the core methods for continuous grooming

Automation

Successful application of Specification by example on large scale projects requires frequent validation of software functionality against a large set of examples. In practice, this requires tests based on examples to be automated. A common approach is to automate the tests but keep examples in a form readable and accessible to non-technical and technical team members, keeping the examples as a single source of truth. This process is supported by a class of test automation tools which work with tests divided into two aspects - the specification and the automation layer. The specification of a test which is often in a plain text or HTML form and contains the examples and auxiliary descriptions. The automation layer connects the example to a software system under test. Examples of such tools are: