Thanks for the answer. Unfortunately, I’m having some issues with your solution.
I just changed GPU and using the same code but with your scaler implementation, I kept getting NaN for the training loss. I thought I had some issues with the new GPU but after a day of experiments I finally realized that if I do not use your scaler, I get the right losses! Moreover, it’s much faster if I do not scale the images (from roughly 19 minutes to 12).