Solved by ptrblck in post #2 
              
              
                We recommend to use the native mixed-precision training utility via torch.cuda.amp instead of apex, as it should cover more tested use cases. 
More information can be found here.