Exactly, in my case, my guess is the following:


I have some embedding tokens that are very very uncommon in the training data (they may even not be present in the whole training dataset).


In this case, the weight decay essentially updates these embedding weights by
w(t+1) =  w(t) - 2  * decayfactor * w(t),
which slowly approaches zero.


When training long enough, these weights become so small that there is some numerical underflow, resulting in NaN.