Solved by ptrblck in post #2 
              
              
                You don’t need to switch to apex/DDP to use automatic mixed-precision. 
We recommend to use the PyTorch implementations of DDP and the native mixed-precision implementation via torch.cuda.amp.