Hi, I use nn.DataParallel to use multi GPU; However it seems that multi-gpu don’t speed up in any aspect.
the details in one epoch are as belowed:
Two GPUs:
Batch_size         Loss   Time(s)
64                 256     320
128                140      324
256                78       330

And Using One GPU:
Batch_size        Loss   Time(s)
64                 250     290
128                137     298
256                77      260

The result really confuses me a lot. so what and where the Multi GPU works for?
In one epoch , the time is almost the same.
PS:
I use the Multi GPU just with nn.Dataparallel
model = model.cuda()
model = nn.Dataparallel(model)


Update:
Multi GPU is for larger batch_size , is it right?  larger batch, faster convergence?