Thank you for your answer. I am using the torch.cuda.amp, so I guess the error is related to the first iterations, as you mention. I haven’t seen invalid outputs or gradients, so I think this isn’t an issue neither.