Yes, there are at least 2 data parallelisms you can do with PyTorch provided classes :


DataParallel(DP)

DistributedDataParallel(DDP), https://pytorch.org/docs/stable/notes/ddp.html


What we discussed before on collecting data across participating processes etc is applied only for DDP. With DP there is a single process running, so there is no need to collect data etc.
However, PyTorch doc suggests to use DDP and it is faster than DP.
PS: See also how ignite simplifies usage of DDP : https://pytorch.org/ignite/distributed.html 1