So, I run following experiments for the comparison. The training set includes 6k images.
I am starting from pretrained resnet18 everytime with a learning rate of 0.0001. I trained upto 15th epochs every time. So, the training accuracy is the final training accuracy after 15th epoch. time_per_epoch is the time it took to train one epoch of data.


Exp 1: Without nn.DataParallel, batch_size = 4. time_per_epoch = 0.47 Min, training_accuracy = 38.82%.


Exp 2: Without nn.DataParallel, batch_size = 8, time_per_epoch = 0.25 Min, training_accuracy = 39.33%.


Exp 3: Without nn.DataParallel, batch_size = 16, time_per_epoch = 0.15 Min,
training_accuracy = 31.26%.


Exp 4: Without nn.DataParallel, batch_size = 32, time_per_epoch = 0.14 Min,
training_accuracy = 17.08%.


Exp 5: Without nn.DataParallel, batch_size = 64, time_per_epoch = 0.13 Min,
training_accuracy = 6.6%


Exp 6: With nn.DataParallel, batch_size = 4, time_per_epoch = 1.27 Min,
training_accuracy = 0.31%.


Exp 7: With nn.DataParallel, batch_size = 8, time_per_epoch = 0.65 Min,
training_accuracy = 9.89%.


Exp 8: With nn.DataParallel, batch_size = 16, time_per_epoch = 0.34 Min,
training_accuracy = 0.14.33%.


Exp 9: With nn.DataParallel, batch_size = 32, time_per_epoch = 0.19 Min,
training_accuracy = 11.19%.


Exp 10: With nn.DataParallel, batch_size = 64, time_per_epoch = 0.13 Min,
training_accuracy = 5.30%.


Exp 11: With nn.DataParallel, batch_size = 128, time_per_epoch = 0.13 Min,
training_accuracy = 1.73%.


From these experiments, I am not able to see the advantage of nn.DataParallel. It seems like it takes longer time than the same counterpart due to data transmission between GPUs. I have no idea on the accuracy difference. But for the batch_size of 64, time_per_epoch and accuracy are comparable.