Solved by miken in post #4 
              
              
                Thanks, @mcarilli. A few iterations in this context is about 1000 iterations. 
And you’re right scaler.scale(loss).backward() should be outside the autocast context. The actual fix was due to how PyTorch did dynamic scaling. For my specific use-case, I had to set the growth_interval parameter to som…