so looking at your code, it looks like you didn’t create the process groups in different processes, and ended up just using one process, or are you just running the scripts on multi hosts? If not, did you try following the DDP tutorial, launch it in multiple processes and see if improves the performance?