If using the recommended DistributedDataParallel (DDP) mode, where there is a dedicated process for each GPU, DDP does not split input data. Each process will have its own data loader and its own DDP instance. DDP only help to automatically compute the global averaged gradient in the backward pass. DDP will run in this mode when device_ids only contains a single device, or there is only one visible device. See this 46 for more details.