Solved by ptrblck in post #2 
              
              
                I wouldn’t assume to see any automatic transformation of gradients if no ddp hooks etc. are used. 
The native mixed-precision training util. via torch.cuda.amp uses FP32 parameters and thus also FP32 gradients. 
You could try to increase the bucket_cap_mb to kick off the gradient sync later, if need…