gil_fernandes:

Could this be the cause for such warnings?


Yes, exactly. That’s what I was trying to say here:



 ptrblck:

If the first iteration creates NaN gradients (e.g. due to a high scaling factor and thus gradient overflow), the optimizer.step() will be skipped and you might get this warning.


Sorry, if I couldn’t describe it clearly. 



 gil_fernandes:

do you get the scale factor using scaler.get_scale() where scaler is an instance of torch.cuda.amp.GradScaler ?


Yes, that’s what I had in mind.