Wow you replied so quickly. Thank you @ptrblck ! What if the first few iterations are not skipped, can I say that using a new scaler is equivalent to increasing the learning rate in the first few iterations? Seems there’s no warning message saying the gradients are scaled in the first few iterations…