nooblyh:

Have you check that the batch sizes are at the same position?


What do you mean by at the same position?
My understanding is that DataParallel takes data of batch size m and splits them as int(m/num_of_devices) sending to each device it’s own split and a copy of the model?