You’re scaling the losses again after the autocast region, which is redundant and might lead to wrong loss I believe.
Remove the scaler.scale(loss) and simply run loss.backward() outside the autocast, that might fix it…
A lot of training will lead to overfitting and negligible loss, but I don’t think it should result in Nan losses