I’m unfortunately not really familiar with Windows and cannot comment on the performance.
However, your point about Dataparallel vs. DDP is a valid concern.
We usually see a better performance with DDP using a single process for each GPU, as the model is replicated only during the start and the GIL shouldn’t get in the way.



 solarflarefx:

I could also experiment with the number of workers, but is there a general guideline on how to adjust this parameter on multiGPU training?


How many workers are you using in each process? If you are using too many workers (processes), you might see an overall slowdown, so you could try to decrease this number and check the performance again. A general guideline is hard to give, as it depends on your used system.