The scatter and gather operations are used, as a “simple” data parallel implementation via nn.DataParallel. This blog post 2 explains the overall workflow in more details.
This overhead is also why we recommend to use DistributedDataParallel with a single process per GPU, which would avoid these copies, and would thus yield the best performance.