Thanks for the response. In my case, I cannot increase the batch size due to memory limitations. However, I’ve noticed that when I evaluate the model trained using DataParallel, every change I make to, e.g., the batch size or the number of gpus, significantly degrade the performance  even when evaluated in train() mode.