Do you have a code snippet that reproduces the issue? From the description I guess most of the profiler results are from moving your model and parameters from CPU to GPU, instead of doing the actual training. Can you profile it after move the model to GPU? The model should be converted to GPU before wrapping it with DDP, see the DDP tutorial 2