I am going to answer my own question. DistributedSampler is not dividing the batch size, only devides the dataset. I would like to ask another question, when the gradients are mixed, I hope that all the processes sends its gradients to the master process, then master process executes the backpropagation and then sends back the new weights. Is it working like this? Because if all the gradients are sended to all processes I will have a peak of memory consumption. Can you help me to clarify this?