Softmax function


In mathematics, the softmax function, also known as softargmax or normalized exponential function, is a function that takes as input a vector z of K real numbers, and normalizes it into a probability distribution consisting of K probabilities proportional to the exponentials of the input numbers. That is, prior to applying softmax, some vector components could be negative, or greater than one; and might not sum to 1; but after applying softmax, each component will be in the interval, and the components will add up to 1, so that they can be interpreted as probabilities. Furthermore, the larger input components will correspond to larger probabilities. Softmax is often used in neural networks, to map the non-normalized output of a network to a probability distribution over predicted output classes.
The standard softmax function is defined by the formula
In words: we apply the standard exponential function to each element of the input vector and normalize these values by dividing by the sum of all these exponentials; this normalization ensures that the sum of the components of the output vector is 1.
Instead of, a different base > 0 can be used; choosing a larger value of will create a probability distribution that is more concentrated around the positions of the largest input values. Writing or yields the expressions:
In some fields, the base is fixed, corresponding to a fixed scale, while in others the parameter is varied.

Interpretations

Smooth arg max

The name "softmax" is misleading; the function is not a smooth maximum, but is rather a smooth approximation to the arg max function: the function whose value is which index has the maximum. In fact, the term "softmax" is also used for the closely related LogSumExp function, which is a smooth maximum. For this reason, some prefer the more accurate term "softargmax", but the term "softmax" is conventional in machine learning. For this section, the term "softargmax" is used to emphasize this interpretation.
Formally, instead of considering the arg max as a function with categorical output , consider the arg max function with one-hot representation of the output :
where the output coordinate if and only if is the arg max of, meaning is the unique maximum value of. For example, in this encoding since the third argument is the maximum.
This can be generalized to multiple arg max values by dividing the 1 between all max args; formally where is the number of arguments assuming the maximum. For example, since the second and third argument are both the maximum. In case all arguments are equal, this is simply Points with multiple arg max values are singular points – these are the points where arg max is discontinuous – while points with a single arg max are known as non-singular or regular points.
With the last expression given in the introduction, softargmax is now a smooth approximation of arg max: as, softargmax converges to arg max. There are various notions of convergence of a function; softargmax converges to arg max pointwise, meaning for each fixed input as, However, softargmax does not converge uniformly to arg max, meaning intuitively that different points converge at different rates, and may converge arbitrarily slowly. In fact, softargmax is continuous, but arg max is not continuous at the singular set where two coordinates are equal, while the uniform limit of continuous functions is continuous. The failure to converge uniformly is because for inputs where two coordinates are almost equal, the arg max is the index of one or the other, so a small change in input yields a large change in output. For example,
but and for all inputs: the closer the points are to the singular set, the slower they converge. However, softargmax does converge compactly on the non-singular set.
Conversely, as, softargmax converges to arg min in the same way, where here the singular set is points with two arg min values. In the language of tropical analysis, the softmax is a deformation or "quantization" of arg max and arg min, corresponding to using the log semiring instead of the max-plus semiring, and recovering the arg max or arg min by taking the limit is called "tropicalization" or "dequantization".
It is also the case that, for any fixed, if one input is much larger than the others relative to the temperature,, the output is approximately the arg max. For example, a difference of 10 is large relative to a temperature of 1:
However, if the difference is small relative to the temperature, the value is not close to the arg max. For example, a difference of 10 is small relative to a temperature of 100:
As, temperature goes to zero,, so eventually all differences become large, which gives another interpretation for the limit behavior.

Probability theory

In probability theory, the output of the softargmax function can be used to represent a categorical distribution – that is, a probability distribution over different possible outcomes.

Statistical mechanics

In statistical mechanics, the softargmax function is known as the Boltzmann distribution : the index set are the microstates of the system; the inputs are the energies of that state; the denominator is known as the partition function, often denoted by ; and the factor is called the coldness.

Applications

The softmax function is used in various multiclass classification methods, such as multinomial logistic regression , multiclass linear discriminant analysis, naive Bayes classifiers, and artificial neural networks. Specifically, in multinomial logistic regression and linear discriminant analysis, the input to the function is the result of distinct linear functions, and the predicted probability for the 'th class given a sample vector and a weighting vector is:
This can be seen as the composition of linear functions and the softmax function. The operation is equivalent to applying a linear operator defined by to vectors, thus transforming the original, probably highly-dimensional, input to vectors in a -dimensional space.

Neural networks

The softmax function is often used in the final layer of a neural network-based classifier. Such networks are commonly trained under a log loss regime, giving a non-linear variant of multinomial logistic regression.
Since the function maps a vector and a specific index to a real value, the derivative needs to take the index into account:
This expression is symmetrical in the indexes and thus may also be expressed as
Here, the Kronecker delta is used for simplicity.
If the function is scaled with the parameter, then these expressions must be multiplied by.
See Multinomial logit for a probability model which uses the softmax activation function.

Reinforcement learning

In the field of reinforcement learning, a softmax function can be used to convert values into action probabilities. The function commonly used is:
where the action value corresponds to the expected reward of following action a and is called a temperature parameter. For high temperatures, all actions have nearly the same probability and the lower the temperature, the more expected rewards affect the probability. For a low temperature, the probability of the action with the highest expected reward tends to 1.

Properties

Geometrically the softmax function maps the vector space to the interior of the standard -simplex, cutting the dimension by one, due to the linear constraint that all output sum to 1 meaning it lies on a hyperplane.
Along the main diagonal softmax is just the uniform distribution on outputs, : equal scores yield equal probabilities.
More generally, softmax is invariant under translation by the same value in each coordinate: adding to the inputs yields, because it multiplies each exponent by the same factor, , so the ratios do not change:
Geometrically, softmax is constant along diagonals: this is the dimension that is eliminated, and corresponds to the softmax output being independent of a translation in the input scores. One can normalize input scores by assuming that the sum is zero, and then the softmax takes the hyperplane of points that sum to zero,, to the open simplex of positive values that sum to 1, analogously to how the exponent takes 0 to 1, and is positive.
By contrast, softmax is not invariant under scaling. For instance, but
The standard logistic function is the special case for a 1-dimensional axis in 2-dimensional space, say the x-axis in the plane. One variable is fixed at 0, so, and the other variable can vary, denote it, so the standard logistic function, and its complement. The 1-dimensional input could alternatively be expressed as the line, with outputs and
The softmax function is also the gradient of the LogSumExp function, a smooth maximum; defining:
the partial derivatives are:
Expressing the partial derivatives as a vector with the gradient yields the softmax.

History

The softmax function was used in statistical mechanics as the Boltzmann distribution in the foundational paper, formalized and popularized in the influential textbook.
The use of the softmax in decision theory is credited to, who used the axiom of independence of irrelevant alternatives in rational choice theory to deduce the softmax in Luce's choice axiom for relative preferences.
In machine learning, the term "softmax" is credited to John S. Bridle in two 1989 conference papers, : and :

Example

If we take an input of , the softmax of that is . The output has most of its weight where the '4' was in the original input. This is what the function is normally used for: to highlight the largest values and suppress values which are significantly below the maximum value. But note: softmax is not scale invariant, so if the input were the softmax would be . This shows that for values between 0 and 1 softmax, in fact, de-emphasizes the maximum value.
Computation of this example using Python code:

>>> import numpy as np
>>> a =
>>> np.exp / np.sum
array

Here is an example of Julia code:

julia> A = ; # semicolon to suppress interactive output
julia> exp../ sum
7-element Array:
0.0236405
0.0642617
0.174681
0.474833
0.0236405
0.0642617
0.174681

Here is an example of R code:

> z <- c
> softmax <- exp/sum
> softmax
0.02364054 0.06426166 0.17468130 0.47483300 0.02364054 0.06426166 0.17468130