False discovery rate


The false discovery rate is a method of conceptualizing the rate of type I errors in null hypothesis testing when conducting multiple comparisons. FDR-controlling procedures are designed to control the expected proportion of "discoveries" that are false. FDR-controlling procedures provide less stringent control of Type I errors compared to familywise error rate controlling procedures, which control the probability of at least one Type I error. Thus, FDR-controlling procedures have greater power, at the cost of increased numbers of Type I errors.

History

Technological motivations

The modern widespread use of the FDR is believed to stem from, and be motivated by, the development in technologies that allowed the collection and analysis of a large number of distinct variables in several individuals. By the late 1980s and 1990s, the development of "high-throughput" sciences, such as genomics, allowed for rapid data acquisition. This, coupled with the growth in computing power, made it possible to seamlessly perform hundreds and thousands of statistical tests on a given data set. The technology of microarrays was a prototypical example, as it enabled thousands of genes to be tested simultaneously for differential expression between two biological conditions.
As high-throughput technologies became common, technological and/or financial constraints led researchers to collect datasets with relatively small sample sizes and large numbers of variables being measured per sample. In these datasets, too few of the measured variables showed statistical significance after classic correction for multiple tests with standard multiple comparison procedures. This created a need within many scientific communities to abandon FWER and unadjusted multiple hypothesis testing for other ways to highlight and rank in publications those variables showing marked effects across individuals or treatments that would otherwise be dismissed as non-significant after standard correction for multiple tests. In response to this, a variety of error rates have been proposed—and become commonly used in publications—that are less conservative than FWER in flagging possibly noteworthy observations.

Literature

The FDR concept was formally described by Yoav Benjamini and Yosef Hochberg in 1995 as a less conservative and arguably more appropriate approach for identifying the important few from the trivial many effects tested. The FDR has been particularly influential, as it was the first alternative to the FWER to gain broad acceptance in many scientific fields. In 2005, the Benjamini and Hochberg paper from 1995 was identified as one of the 25 most-cited statistical papers.
Prior to the 1995 introduction of the FDR concept, various precursor ideas had been considered in the statistics literature. In 1979, Holm proposed the Holm procedure, a stepwise algorithm for controlling the FWER that is at least as powerful as the well-known Bonferroni adjustment. This stepwise algorithm sorts the p-values and sequentially rejects the hypotheses starting from the smallest p-values.
Benjamini said that the false discovery rate, and the paper Benjamini and Hochberg, had its origins in two papers concerned with multiple testing:
The BH procedure was proven to control the FDR for independent tests in 1995 by Benjamini and Hochberg. In 1986, R. J. Simes offered the same procedure as the "Simes procedure", in order to control the FWER in the weak sense when the statistics are independent.

Definitions

Based on definitions below we can define as the proportion of false discoveries among the discoveries :
where is the number of false discoveries and is the number of true discoveries.
The false discovery rate is then simply:
where is the expected value of. The goal is to keep FDR below a given threshold q. To avoid division by zero, is defined to be 0 when. Formally,.

Classification of multiple hypothesis tests

Controlling procedures

The settings for many procedures is such that we have null hypotheses tested and their corresponding p-values. We list these p-values in ascending order and denote them by. A procedure that goes from a small p-value to a large one will be called a step-up procedure. In a similar way, in a "step-down" procedure we move from a large corresponding test statistic to a smaller one.

Benjamini–Hochberg procedure

The Benjamini–Hochberg procedure controls the FDR at level. It works as follows:
  1. For a given, find the largest such that
  2. Reject the null hypothesis for all for.
Geometrically, this corresponds to plotting vs. , drawing the line through the origin with slope , and declaring discoveries for all points on the left up to and including the last point that is below the line.
The BH procedure is valid when the tests are independent, and also in various scenarios of dependence, but is not universally valid. It also satisfies the inequality:
If an estimator of is inserted into the BH procedure, it is no longer guaranteed to achieve FDR control at the desired level. Adjustments may be needed in the estimator and several modifications have been proposed.
Note that the mean for these tests is, the Mean or MFDR, adjusted for independent or positively correlated tests. The MFDR expression here is for a single recomputed value of and is not part of the Benjamini and Hochberg method.

Benjamini–Yekutieli procedure

The Benjamini–Yekutieli procedure controls the false discovery rate under arbitrary dependence assumptions. This refinement modifies the threshold and finds the largest such that:
In the case of negative correlation, can be approximated by using the Euler–Mascheroni constant.
Using MFDR and formulas above, an adjusted MFDR, or AFDR, is the min for dependent tests.
The other way to address dependence is by bootstrapping and rerandomization.

Properties

Adaptive and scalable

Using a multiplicity procedure that controls the FDR criterion is adaptive and scalable. Meaning that controlling the FDR can be very permissive, or conservative - all depending on the number of hypotheses tested and the level of significance.
The FDR criterion adapts so that the same number of false discoveries will have different implications, depending on the total number of discoveries. This contrasts with the family wise error rate criterion. For example, if inspecting 100 hypotheses :
The FDR criterion is scalable in that the same proportion of false discoveries out of the total number of discoveries, remains sensible for different number of total discoveries. For example:
Controlling the FDR using the linear step-up BH procedure, at level q, has several properties related to the dependency structure between the test statistics of the null hypotheses that are being corrected for. If the test statistics are:
If all of the null hypotheses are true, then controlling the FDR at level guarantees control over the FWER : , simply because the event of rejecting at least one true null hypothesis is exactly the event, and the event is exactly the event . But if there are some true discoveries to be made then. In that case there will be room for improving detection power. It also means that any procedure that controls the FWER will also control the FDR.

Related concepts

The discovery of the FDR was preceded and followed by many other types of error rates. These include:
The false coverage rate is, in a sense, the FDR analog to the confidence interval. FCR indicates the average rate of false coverage, namely, not covering the true parameters, among the selected intervals. The FCR gives a simultaneous coverage at a level for all of the parameters considered in the problem. Intervals with simultaneous coverage probability 1−q can control the FCR to be bounded by q. There are many FCR procedures such as: Bonferroni-Selected–Bonferroni-Adjusted, Adjusted BH-Selected CIs, Bayes FCR, and other Bayes methods.

Bayesian approaches

Connections have been made between the FDR and Bayesian approaches, thresholding wavelets coefficients and model selection, and generalizing the confidence interval into the false coverage statement rate.

False positive rates in single tests of significance

Colquhoun used the term "false discovery rate" to mean the probability that a statistically significant result was a false positive. This was part of an investigation of the question "how should one interpret the P value found in a single unbiased test of significance". In subsequent work, Colquhoun called the same thing the false positive risk, rather than the false discovery rate in order to avoid confusion with the use of the latter term in connection with the problem of multiple comparisons. The methods for dealing with multiple comparisons described above aim to control the type 1 error rate. The result of applying them is to produce a P value. The result is, therefore, subject to the same misinterpretations as any other P value.