The code looks generally alright.
Have you tried to reduce the number of workers? 20 seems to be quite high (of course it’s depending on your system setup).
Also, could you check if DistributedDataParallel 8 speeds up your training?