I am training a model on top of ALBERT using mixed precision with apex. And the loss keeps decreasing like this 
image735×265 49.6 KB
. How can i track the problem? I found there was “nans” in the matrix and i fixed it with LayerNorm as they tend to be very large.