One option might be using nvprof and then visualize the result. It will show time consumed by different comp and comm ops. See the following links:

https://www.telesens.co/2019/04/04/distributed-data-parallel-training-using-pytorch-on-aws/ 2
https://developer.nvidia.com/blog/cuda-pro-tip-nvprof-your-handy-universal-gpu-profiler/ 1

We are also working on extending autograd profiler to work with DDP, but we don’t have a target date for it yet.