The soft maximum often comes up when dealing with probabilistic models and in neural networks. I'm surprised that a blog that has been highlighting numerical issues hasn't pointed out how the soft maximum should be <i>computed</i>.<p>If x is vector of values then the naive code is:<p><pre><code>  log(sum(exp(x)))
</code></pre>
where exp operates elementwise. However, if even a single item of x is large (1000 say) this will return Inf. If x are log probabilities, where you want the log of the sum of the probabilities (common), the elements of x might all be less than -1000 (also common) and then the function will return -Inf.<p>Many people have a function called logsumexp() kicking about to compute the softmax robustly. It will do something like:<p><pre><code>    y = max(x);
    return y + log(sum(exp(x-y)));
</code></pre>
One example, slightly more elaborate, Matlab implementation is in:
<a href="http://research.microsoft.com/en-us/um/people/minka/software/lightspeed/" rel="nofollow">http://research.microsoft.com/en-us/um/people/minka/software...</a>