oh, my bad. I got confused a whole lot because all my life I am used to seeing mini-batch times. Sorry for the misunderstanding @mattmacy , I apologize.
But as @ngimel and @apaszke pointed out, there are many scenarios in which DataParallel is not great, especially if you have too little compute or if you have too many parameters in your model.