Hi I am training a Bert model with amp O2 using 64 gpus . It trains successfully for 217 epochs but at 218th epoch, the loss scale goes down rapidly and model fails with division by zero error. Run does not fail when apex is disabled.
What can be the problem here?
Printed the gradients and Max param after backward . Max param is 11, gradients are Nan. Why does Nan appear ?