Hi, one issue I see is that you are spawning 16 data reading workers on each distributed process for a total of 32 processes just for data reading, do you need that many workers? This might be creating too many processes on your machine resulting in trashing/slowdown. Could you try setting num_workers=0 and comparing the training speed of local vs distributed (ensuring the distributed training splits the data, which it appears it does correctly in the script)?