Hi Mukesh!



 Mukesh1729:


When I add the softmax the network loss doesn’t decrease and is around the same point and works when I remove the softmax.



As a practical matter, I would suggest not using softmax() for the
simple reason that your network trains better without it than with it.
Most of what we know about what works with neural networks comes
not from theory or mathematical proofs, but from experience.  So go
ahead and do what your experience with your use case shows works.
I can give some hand-waving intuition about why softmax() might
not be a good fit with mse_loss().  The “mse” in mse_loss() stands
for “mean-squared-error.”  Roughly speaking, this is the variance of
the mismatch between your predictions and targets (and the variance
is the square of the standard deviation).  Standard deviations and
variances “naturally” work in a context where the values involved are
unconstrained and run from -inf to inf.  Your softmax() takes
such unbounded values and forces them to be probability-like values
that range over [0.0, 1.0].
This doesn’t mean that using softmax() couldn’t work or be a good
choice for your use case, but this intuition suggests that softmax()
isn’t a natural fit when using mse_loss().
Best.
K. Frank.