as adam pointed out, DataParallel is very very well defined in Deep Learning. We split the mini-batches over multiple GPUs, and accumulate the gradients at the end from all the GPUs before doing the optimization step.
Getting a linear speedup when doubling the batch size is the best case scenario, and you are hitting that.
If there was a confusion in terminology, i hope it’s clarified now.