Were the training jobs with and without Apex done on the same machines? It seems like GPU-GPU communication is working based on the Ring/Trees logs that you pasted. It might make sense to direct this issue to the Apex GitHub repo since the training is working with vanilla DDP (we’ve also been working on bridging the gap between vanilla DDP and Apex through new features like dynamic bucketing, so the performance difference may not be as much as before).