@ekurtic Thanks for quick response.
I am trying to find how is the gain in training time (without affecting accuracy as much) by using DistributedDataParallel. As per you, if I use batch size of 4, then yes the training will be faster. However in that case I will be using LR of batch size 32 for that of 4. Is this okay? And if I divide LR by 8, then I am effectively slowing down the training, and thus not gaining on training time. Am I missing something here?