Solved by ptrblck in post #2 
              
              
                The apex implementations are deprecated, since they are now supported in PyTorch via their native implementations, so you should not use apex/DDP or apex/AMP anymore. 
This post explains it in more detail.