Imprecise Dirichlet process


In probability theory and statistics, the Dirichlet process is one of the most popular Bayesian nonparametric models. It was
introduced by Thomas Ferguson as a prior over probability distributions.
A Dirichlet process is completely defined by its parameters: is an arbitrary distribution and is a positive real number.
According to the Bayesian paradigm these parameters should be chosen based on the available prior information on the domain.
The question is: how should we choose the prior parameters of the DP, in particular the infinite dimensional one, in case of lack of prior information?
To address this issue, the only prior that has been proposed so far is the limiting DP obtained for, which has been introduced under
the name of Bayesian bootstrap by Rubin; in fact it can be proven that the Bayesian bootstrap is asymptotically equivalent to the frequentist bootstrap introduced by Bradley Efron.
The limiting Dirichlet process has been criticized on diverse grounds. From an a-priori point of view, the main
criticism is that taking is far from leading to a noninformative prior.
Moreover, a-posteriori, it assigns zero probability to any set that does not include the observations.
The imprecise Dirichlet process has been proposed to overcome these issues. The basic idea is to fix but do not choose any precise base measure.
More precisely, the imprecise Dirichlet process is defined as follows:
where is the set of all probability measures. In other words, the IDP is the set of all Dirichlet processes obtained
by letting the base measure to span the set of all probability measures.

Inferences with the Imprecise Dirichlet Process

Let a probability distribution on and assume that.
Then consider a real-valued bounded function defined on. It is well known that the expectation of with respect to the Dirichlet process is
One of the most remarkable properties of the DP priors is that the posterior distribution of is again a DP.
Let be an independent and identically distributed sample from and, then the posterior distribution of given the observations is
where is an atomic probability measure centered at. Hence, it follows
that
Therefore, for any fixed, we can exploit the previous equations to derive prior and posterior expectations.
In the IDP can span the set of all distributions. This implies that we will get a different prior and posterior expectation of for any choice of . A way to characterize inferences for the IDP is by computing lower and upper bounds for the expectation of w.r.t..
A-priori these bounds are:
the lower bound is obtained by a probability measure that puts all the mass on the infimum of, i.e., with . From the above expressions of the lower and upper bounds, it can be observed that the range of under the IDP is the same as the original range of. In other words, by specifying the IDP, we are not giving any prior information on the value of the expectation of. A-priori, IDP is therefore a model of prior -ignorance for.
A-posteriori, IDP can learn from data. The posterior lower and upper bounds for the expectation of are in fact given by:
It can be observed that the posterior inferences do not depend on . To define the IDP, the modeler has only to choose . This explains the meaning of the adjective near in prior near-ignorance, because the IDP requires by the modeller the elicitation of a parameter. However, this is a simple elicitation problem for a nonparametric prior, since we only have to choose the value of a positive scalar.
Finally, observe that for, IDP satisfies
where. In other words, the IDP is consistent.

Choice of the prior strength s

The IDP is completely specified by, which is the only parameter left in the prior model.
Since the value of determines how quickly lower and upper posterior expectations converge at the
increase of the number of observations, can be chosen so to match a certain convergence rate.
The parameter can also be chosen to have some desirable frequentist properties, see

Example: estimate of the cumulative distribution

Let be i.i.d. real random variables with cumulative distribution function.
Since, where is the indicator function, we can use
IDP to derive inferences about The lower and upper posterior mean of are
where is the empirical distribution function. Here, to obtain the lower we have exploited the fact that and for the upper that.
File:Beta distribution for the lower and upper probability of the hypothesis..jpeg|thumb|Beta distributions for the lower and upper probability corresponding to the observations. The area in gives
the lower and the upper probability of the hypothesis "the median is greater than zero".
Note that, for any precise choice of , the posterior expectation of will be included between the lower and upper bound.

Example: median test

IDP can also be used for hypothesis testing, for instance to test the hypothesis , i.e., the median of is greater than zero.
By considering the partition and the property of the Dirichlet process, it can be shown that
the posterior distribution of is
where is the number of observations that are less than zero,
By exploiting this property, it follows that
where is the regularized incomplete beta function.
We can thus perform the hypothesis test
and then
  1. if both the inequalities are satisfied we can declare that with probability larger than ;
  2. if only one of the inequality is satisfied, we are in an indeterminate situation, i.e., we cannot decide;
  3. if both are not satisfied, we can declare that the probability that is lower than the desired probability of.
IDP returns an indeterminate decision when the decision is prior dependent.
By exploting the relationship between the cumulative distribution function of the Beta distribution, and the cumulative distribution function of a random variable Z from a binomial distribution, where the "probability of success" is p and the sample size is n:
we can show that the median test derived with th IDP for any choice of encompasses the one-sided frequentist
sign test as a test for the median. It can in fact be verified that for the -value of the sign test is equal to. Thus, if then the -value is less than and, thus, they two tests have the same power.

Applications of the Imprecise Dirichlet Process

Dirichlet processes are frequently used in Bayesian nonparametric statistics. The Imprecise Dirichlet Process
can be employed instead of the Dirichlet processes in any application in which prior information is lacking.
In this respect, the Imprecise Dirichlet Process has been used for nonparametric hypothesis testing, see .
Based on the Imprecise Dirichlet Process, Bayesian nonparametric near-ignorance versions of the following classical nonparametric estimators
have been derived: the Wilcoxon rank sum test and the Wilcoxon signed-rank test.
A Bayesian nonparametric near-ignorance model presents several advantages with respect to a traditional approach to hypothesis testing.
  1. The Bayesian approach allows us to formulate the hypothesis test as a decision problem. This means that we can verify the evidence in favor of the null hypothesis and not only rejecting it and take decisions which minimize the expected loss.
  2. Because of the nonparametric prior near-ignorance, IDP based tests allows us to start the hypothesis test with very weak prior assumptions, much in the direction of letting data speak for themselves.
  3. Although the IDP test shares several similarities with a standard Bayesian approach, at the same time it embodies a significant change of paradigm when it comes to take decisions. In fact the IDP based tests have the advantage of producing an indeterminate outcome when the decision is prior-dependent. In other words, the IDP test suspends the judgment when the option which minimizes the expected loss changes depending on the Dirichlet Process base measure we focus on.
  4. It has been empirically verified that when the IDP test is indeterminate, the frequentist tests are virtually behaving as random guessers. This surprising result has practical consequences in hypothesis testing. Assume that we are trying to compare the effects of two medical treatments and that, given the available data, the IDP test is indeterminate. In such a situation the frequentist test always issues a determinate response, but it turns out that its response is completely random, like if we were tossing of a coin. On the other side, the IDP test acknowledges the impossibility of making a decision in these cases. Thus, by saying "I do not know", the IDP test provides a richer information to the analyst. The analyst could for instance use this information to collect more data.

    Categorical variables

For categorical variables, i.e., when has a finite number of elements, it is known that
the Dirichlet process reduces to a Dirichlet distribution.
In this case, the Imprecise Dirichlet Process reduces to the Imprecise Dirichlet model proposed by Walley as a model for prior -ignorance for chances.