Test and learn


Test and Learn is a set of practices followed by retailers, banks and other consumer-focused companies to test ideas in a small number of locations or customers to predict impact. The process is often designed to answer three questions about any tested program before rollout:
  1. What impact will the program have on key performance indicators if executed across the network or customer base?
  2. Will the program have a larger impact on some stores/customers than others?
  3. Which components of the idea are actually working?

    Historical context

Test and Learn has been systematically applied as far back as 1988 by Capital One. Capital One has been aggressive about testing since the firm was founded, testing everything from product design to marketing to customer selection to collection policies. In a single year, the company performs tens of thousands of tests, allowing it to offer thousands of different types of credit cards to customers, based on the knowledge gained from their tests. Richard Fairbank, CEO of Capital One, called test and learn "a marketing revolution that can be applied to many businesses".

Examples

There are a variety of software tools available today to support systematic testing within an organization. However, there is currently no one software package that covers all types of tests, and in some cases significant knowledge of statistics is still required for effective analysis.
Many companies, including Capital One and eBay, have developed experiment management software within the company. Alternatives to internal solutions include broad statistical analysis software tools such as SAS or test and learn focused software from Applied Predictive Technologies and MarketDial. There are also a variety of more narrowly tailored products, focused on particular problems or fields. In web analytics, for example, firms such as Omniture, WebTrends, and Google have created specialized software.

Criticism

Companies wanting to conduct Test and Learn must mobilize, track and analyze site and customer data at very granular levels. This can often be a massive and expensive undertaking. Increasingly however, an exponential increase in computing power, and corresponding decrease in its cost, have made testing more accessible. In the Harvard Business Review February 2009 article "How to Design Smart Business Experiments", Thomas Davenport discusses the Test and Learn Process, and describes how several companies have overcome these and other barriers to testing.