Let be a continuously-differentiable, strictly convex function defined on a closed convex set. The Bregman distance associated with F for points is the difference between the value of F at point p and the value of the first-order Taylor expansion of F around point q evaluated at point p:
Properties
Non-negativity: for all p, q. This is a consequence of the convexity of F.
Convexity: is convex in its first argument, but not necessarily in the second argument
Linearity: If we think of the Bregman distance as an operator on the function F, then it is linear with respect to non-negative coefficients. In other words, for strictly convex and differentiable, and,
Duality: The function F has a convex conjugate. The Bregman distance defined with respect to has an interesting relationship to
Mean as minimizer: A key result about Bregman divergences is that, given a random vector, the mean vector minimizes the expected Bregman divergence from the random vector. This result generalizes the textbook result that the mean of a set minimizes total squared error to elements in the set. This result was proved for the vector case by, and extended to the case of functions/distributions by. This result is important because it further justifies using a mean as a representative of a random set, particularly in Bayesian estimation.
The squared Mahalanobis distance, which is generated by the convex function. This can be thought of as a generalization of the above squared Euclidean distance.
A key tool in computational geometry is the idea of projective duality, which maps points to hyperplanes and vice versa, while preserving incidence and above-below relationships. There are numerous analytical forms of the projective dual: one common form maps the point to the hyperplane. This mapping can be interpreted as the convex conjugate mapping that takes the point p to its dual point, where F defines the d-dimensional paraboloid. If we now replace the paraboloid by an arbitrary convex function, we obtain a different dual mapping that retains the incidence and above-below properties of the standard projective dual. This implies that natural dual concepts in computational geometry like Voronoi diagrams and Delaunay triangulations retain their meaning in distance spaces defined by an arbitrary Bregman divergence. Thus, algorithms from "normal" geometry extend directly to these spaces
Generalization of Bregman divergences
Bregman divergences can be interpreted as limit cases of skewed Jensen divergences. Jensen divergences can be generalized using comparative convexity, and limit cases of these skewed Jensen divergences generalizations yields generalized Bregman divergence. The Bregman chord divergence is obtained by taking a chord instead of a tangent line.
Bregman divergence on other objects
Bregman divergences can also be defined between matrices, between functions, and between measures. Bregman divergences between matrices include the Stein's loss and von Neumann entropy. Bregman divergences between functions include total squared error, relative entropy, and squared bias; see the references by Frigyik et al. below for definitions and properties. Similarly Bregman divergences have also been defined over sets, through a submodular set function which is known as the discrete analog of a convex function. The submodular Bregman divergences subsume a number of discrete distance measures, like the Hamming distance, precision and recall, mutual information and some other set based distance measures For a list of common matrix Bregman divergences, see Table 15.1 in.
Applications
In machine learning, Bregman divergences are used to calculate the bi-tempered logistic loss, performing better than the softmax function with noisy datasets..
