The perplexity PP of a discrete probability distributionp is defined as where H is the entropy of the distribution and x ranges over events. This measure is also known in some domains as the diversity. Perplexity of a random variableX may be defined as the perplexity of the distribution over its possible values x. In the special case where p models a fair k-sided die, its perplexity is k. A random variable with perplexity k has the same uncertainty as a fair k-sided die, and one is said to be "k-ways perplexed" about the value of the random variable. Perplexity is sometimes used as a measure of how hard a prediction problem is. This is not always accurate. If you have two choices, one with probability 0.9, then your chances of a correct guess are 90 percent using the optimal strategy. The perplexity is 2−0.9 log2 0.9 - 0.1 log2 0.1= 1.38. The inverse of the perplexity, is 1/1.38 = 0.72, not 0.9. The perplexity is the exponentiation of the entropy, which is a more clearcut quantity. The entropy is a measure of the expected, or "average", number of bits required to encode the outcome of the random variable, using a theoretical optimal variable-length code, cf. It can equivalently be regarded as the expected information gain from learning the outcome of the random variable.
A model of an unknown probability distribution p, may be proposed based on a training sample that was drawn from p. Given a proposed probability model q, one may evaluate q by asking how well it predicts a separate test sample x1, x2,..., xN also drawn from p. The perplexity of the model q is defined as where is customarily 2. Better models q of the unknown distribution p will tend to assign higher probabilities q to the test events. Thus, they have lower perplexity: they are less surprised by the test sample. The exponent above may be regarded as the average number of bits needed to represent a test event xi if one uses an optimal code based on q. Low-perplexity models do a better job of compressing the test sample, requiring few bits per test element on average because q tends to be high. The exponent may also be regarded as a cross-entropy, where denotes the empirical distribution of the test sample.
Perplexity per word
In natural language processing, perplexity is a way of evaluating language models. A language model is a probability distribution over entire sentences or texts. Using the definition of perplexity for a probability model, one might find, for example, that the average sentence xi in the test sample could be coded in 190 bits. This would give an enormous model perplexity of 2190 per sentence. However, it is more common to normalize for sentence length and consider only the number of bits per word. Thus, if the test sample's sentences comprised a total of 1,000 words, and could be coded using a total of 7.95 bits per word, one could report a model perplexity of 27.95 = 247 per word. In other words, the model is as confused on test data as if it had to choose uniformly and independently among 247 possibilities for each word. The lowest perplexity that has been published on the Brown Corpus as of 1992 is indeed about 247 per word, corresponding to a cross-entropy of log2247 = 7.95 bits per word or 1.75 bits per letter using a trigram model. It is often possible to achieve lower perplexity on more specialized corpora, as they are more predictable. Again, simply guessing that the next word in the Brown corpus is the word "the" will have an accuracy of 7 percent, not 1/247 = 0.4 percent, as a naive use of perplexity as a measure of predictiveness might lead one to believe. This guess is based on the unigram statistics of the Brown corpus, not on the trigram statistics, which yielded the word perplexity 247. Using trigram statistics would further improve the chances of a correct guess.
