Hello Umang!



 Umang_Sharma:

However, I am still a bit confused as to why the loss gave values of such different magnitudes just by introducing the numerically stable log-sum-exp.


This is just speculation on my part.  I don’t have a clear picture of
the exact sequence of events, but it sounded to me like when you
used the naive log-sum-exp, your training broke early with NaNs.
And when you used the more stable log-sum-exp, your training
gave very large loss values, and then later broke with NaNs.
I assume that the naive version broke before you got to the large
loss values, so they were “hidden.”  The stable version was able
to run further, so it was able to reach the large loss values.  (Again,
just speculation).
Best.
K. Frank