If the program uses GPUs, you can use elapsed_time 22 to measure the time spent on forward, backward, and optimizer. It is harder to break down computation and communication of the backward pass, as DDP tries to overlap these two and DDP conducts communication on dedicated CUDA streams that are not visible from the application side. Besides, communications are launched as soon as a gradient bucket is ready, meaning that it may or may not always saturate the bandwidth. To get around this, you can run local forward-backward, and then explicitly using allreduce from application side to conduct gradient synchronization after the backward pass. This will expose opportunities to measure that from application.