Smooth approximation to the maximum function
The LogSumExp (LSE) (also called RealSoftMax or multivariable softplus) function is a smooth maximum – a smooth approximation to the maximum function, mainly used by machine learning algorithms. It is defined as the logarithm of the sum of the exponentials of the arguments:
formula_1
Properties.
The LogSumExp function domain is formula_2, the real coordinate space, and its codomain is formula_3, the real line. 
It is an approximation to the maximum formula_4 with the following bounds
formula_5
The first inequality is strict unless formula_6. The second inequality is strict unless all arguments are equal.
In addition, we can scale the function to make the bounds tighter. Consider the function formula_9. Then 
formula_10
(Proof: Replace each formula_11 with formula_12 for some formula_13 in the inequalities above, to give
formula_14
and, since formula_13
formula_16
finally, dividing by formula_17 gives the result.)
Also, if we multiply by a negative number instead, we of course find a comparison to the formula_18 function:
formula_19
The LogSumExp function is convex, and is strictly increasing everywhere in its domain. It is not strictly convex, since it is affine (linear plus a constant) on the diagonal and parallel lines:
formula_20
Other than this direction, it is strictly convex (the Hessian has rank &amp;NoBreak;&amp;NoBreak;), so for example restricting to a hyperplane that is transverse to the diagonal results in a strictly convex function. See formula_21, below.
Writing formula_22 the partial derivatives are:
formula_23
which means the gradient of LogSumExp is the softmax function.
The convex conjugate of LogSumExp is the negative entropy.
log-sum-exp trick for log-domain calculations.
The LSE function is often encountered when the usual arithmetic computations are performed on a logarithmic scale, as in log probability.
Similar to multiplication operations in linear-scale becoming simple additions in log-scale, an addition operation in
linear-scale becomes the LSE in log-scale:
formula_24
A common purpose of using log-domain computations is to increase accuracy and avoid underflow and overflow problems
when very small or very large numbers are represented directly (i.e. in a linear domain) using limited-precision
floating point numbers.
Unfortunately, the use of LSE directly in this case can again cause overflow/underflow problems. Therefore, the
following equivalent must be used instead (especially when the accuracy of the above 'max' approximation is not sufficient).
formula_25
where formula_26
Many math libraries such as IT++ provide a default routine of LSE and use this formula internally.
A strictly convex log-sum-exp type function.
LSE is convex but not strictly convex.
We can define a strictly convex log-sum-exp type function by adding an extra argument set to zero:
formula_27
This function is a proper Bregman generator (strictly convex and differentiable). 
It is encountered in machine learning, for example, as the cumulant of the multinomial/binomial family.
In tropical analysis, this is the sum in the log semiring.
References.
 ;templatestyles src="Reflist/styles.css" / ;
 ;templatestyles src="Refbegin/styles.css" / ;