We recommend to use the native DDP implementation instead of apex/DDP, as it’ll be further developed and improved. Could you switch and rerun your code?
Also, use torch.cuda.amp instead of apex/amp for the same reason.