in my opinion,  when you change the batch size,  the learning rate should be scaled up or down proportionally. Can you try this experiment? Batch size 64, without DataParallel,  let’s say learning rate = 0.01