We generally recommend using nn.DistributedDataParallel using a single process for each GPU to avoid the communication overhead from nn.DataParallel.
Could you try that and check the performance again?