Hi Raghul and Chunchun!



 chunchun:

I have no idea if I seperately use the softmax and log function instead of log_softmax,whether there will be a problem when BP.





 mailcorahul:

I don’t think it will cause any problem. It’s still same as using log_softmax.


Just to clarify:
log (softmax()) is mathematically the same as log_softmax(),
but they differ numerically.  softmax() calculates exponentials that
can “blow numbers up.”  The log() then undoes this, but the damage
can already be done.  So log (softmax()) can be numerically unstable,
leading to reduced precision and nans, and can cause problems.
log_softmax() (largely) avoids this by reorganizing the calculation
so that the intermediate blow-up doesn’t occur.  (That’s why pytorch
(and other packages) include it as separate function.)
There is usually no reason to use softmax().  Just feed the last linear
layer of your network (that you would have fed into softmax()) into
cross_entropy() as your loss function (or use log_softmax()
followed by nll_loss()).
If somebody forces you to use softmax() then you’re stuck, and
have to deal with the potential numerical instability of of softmax()
followed by log() …
Good luck!
K. Frank