Hi,
I’ve got a network containing:
Input → LayerNorm → LSTM → Relu → LayerNorm → Linear → output
With gradient clipping set to a value around 1. After the first training epoch, I see that the input’s LayerNorm’s grads are all equal to NaN, but the input in the first pass does not contain NaN or Inf so I have no idea why this is happening or how to prevent it from happening. If I try removing some components (e.g. the second LayerNorm) the NaN grad just moves to another component (such as Linear.bias).
The only thing that comes to mind is that I’ve got AMP mode enabled. After the first iteration the output gets mapped by a loss function to a value around ~100. During the backward pass AMP scales this loss value up to ~600,000. Is it possible that this value is propagated backward and eventually causes the LayerNorm’s gradients to become NaN?
I’ve seen similar errors occurring even with AMP disables so I’m wondering: big picture, how am I supposed to guard against this? I mean, it doesn’t sound like I’m doing anything wrong per-se. The inputs are valid. I’ve got gradient clipping enabled. And yet this error still occurs.
Any ideas?