Hi @InnovArul,
Thank you for the suggestion. So learning rate seems to be the issue. I did three experiments and here’s the result after 15th epochs.


Exp 1: Without nn.DataParallel, learning_rate = 0.01, batch_size = 64. time_per_epoch = 0.13 Min, train_acc = 75.12%.


Exp 2: With nn.DataParallel, learning_rate = 0.01, batch_size = 64, time_per_epoch = 0.13 Min, train_acc = 69.16%.


Exp 3: With nn.DataParallel, learning_rate = 0.01, batch_size = 128, time_per_epoch = 0.13 Min, train_acc = 72.45%.


The accuracies seems comparable now. But with batch_size=128, it takes same time per epoch as of batch_size=64. And I think nn.DataParallel loads every data to gpu:0 first, that may be the reason my system crashes if I put batch_size of 256. Do you have any solution for this? Thanks for the reply.