LogSumExp


The LogSumExp function is a smooth maximum – a smooth approximation to the maximum function, mainly used by machine learning algorithms. It is defined as the logarithm of the sum of the exponentials of the arguments:
In tropical analysis, this is the sum in the log semiring.

Properties

The LogSumExp function domain is, the real coordinate space, and its range is, the real line. The larger the values of or their deviation, the better the approximation becomes. The LogSumExp function is convex, and is strictly monotonically increasing everywhere in its domain.
LSE is a smooth maximum because, applying the tangent line approximation if one term, is much larger than the rest, the second term is small because it has in the denominator, and one gets:
Indeed, there are the following tight bounds :
The upper bound is equality if and only if all are equal.
This is because , and for positive numbers, for any term, include the maximum, and in fact is strict if . Combining with logarithms and exponents, one gets:
The lower bound is met only for, otherwise it is strict but approached when all but one of the arguments approach negative infinity, and the upper bound is met when all the arguments are equal.
Writing the partial derivatives are:
This can be calculated via logarithmic differentiation.
Expressing the partial derivatives as a vector with the gradient yields the softmax function, the multivariable analog of the logistic function.
The convex conjugate of LogSumExp is the negative entropy.

log-sum-exp trick for log-domain calculations

The LSE function is often encountered when the usual arithmetic computations are performed on a logarithmic scale, as in log probability.
Similar to multiplication operations in linear-scale becoming simple additions in log-scale, an addition operation in
linear-scale becomes the LSE in log-scale.
A common purpose of using log-domain computations is to increase accuracy and avoid underflow and overflow problems
when very small or very large numbers are represented directly using limited-precision
floating point numbers.
Unfortunately, the use of LSE directly in this case can again cause overflow/underflow problems. Therefore, the
following equivalent must be used instead.
Therefore, many math libraries such as IT++ provide a default routine of LSE and use this formula internally.
where

A strictly convex log-sum-exp type function

LSE is convex but not strictly convex.
We can define a strictly convex log-sum-exp type function by adding an extra argument set to zero:
This function is a proper Bregman generator.
It is encountered in machine learning, for example, as the cumulant of the multinomial/binomial family.