nn.DistributedDataParallel with a single process per GPU is the recommended way as it’ll be faster than nn.DataParallel.
To increase the data loading performance, play around with the number of workers and select the “sweet spot” for your system. Also make sure the data is stored locally (on an SSD) and not loaded from the network or a HDD.