Should the device be cuda:0 or cuda.
Also, after using multiple gpus and dataparallel, we should be able to increase the batch size as compared to a single gpu. In my case I am still not able to increase the batch size.