Solved by mcarilli in post #4 
              
              
                scaler.unscale_(optimizer) unscales the .grad attributes of all params owned by optimizer, after those .grads have been fully accumulated for those parameters this iteration and are about to be applied.  If you intend to accumulate more gradients into .grads later in the iteration, scaler.unscale_ i…