I am wondering why the  nn.dataparallel which use 3 gpus is slower than a single gpu. The batch size is 50. A single gpu uses 1 minutes for a epoch, while the dataparallel uses 3 minutes for a epoch.
The way I am using dataparallel is
net = Net()
net = torch.nn.DataParallel(net, device_ids=[0, 1, 2]).cuda()
Is the way I am using dataparallel? Thanks.