But I’ve checked and the batch size is not reduced. So if I set the batch size to 256 I would expect to see 64 to each GPU. But instead I see 256. And by reading the code it seems that all processes will get the exact same input especially if you fix the seed in the beginning with torch.seed (which you should do otherwise the models don’t start from the same weights)