exp(log_softmax) vs softmax as neural network activation I have read about log_softmax being more numerical stable than softmax, since it circumvents the division. I need to use softmax, probabilities between 0 and 1, for my neural network loss function. So I have been wondering:
Should one use exp(log_softmax) or softmax as activation function for the output layer?

Numerical stability and relisience to underflow is preserved by undertaking all intermediate computations in log-space to avoid the loss of small numbers due to underflow (i.e., rounding small numbers down to zero).  When you are working in log-space you would use the logsoftmax function, but in some cases you wish to convert back to regular space at the end of your computation, in which case you would convert back via exponentiation.  Ultimately, numerical stability and relisience to underflow is preserved by ensuring that intermediate steps in your code involving small numbers are done in log-space, and the resulting outputs are converted back to regular space only at the end of the computation.  This context will inform when it is appropriate to use each of these functions.
