Thank you for your reply. After reading your answer, I have understood DataParallel but still confused by Distributed DataParallel. In my case, I have one machine with 4 GPUs. According to pytorch 1.0 tutorial , distributed dataparallel can also be used in single machine. Now I have 4 processes , each process has one GPU. So if batch_size is set to 128, that means each process (or single GPU) will be allocated batch_size 32 ? And some hyperparameters like LR should be set with batchsize 32 ?