This does not work for CUDA ops, as they return immediately after the op inserted into the CUDA stream before they are actually done.

Ok, I didn’t know the detail that you explained, but I meant to do a rough estimate so I thought that’s good enough. Nevertheless, the result matches the feeling or actual time spent for one epoch of training, as you can see if you run it. The DDP program takes 47 sec. while DP takes 32 sec. in my environment.