ptrblck:

It might depend how you are using  DistributedDataParallel .


Actually I did not use DistributedDataParallel, we are still using one machine with single GPU but 24 cores of CPU, right now we don’t think GPU is bottleneck, since CPU may not able to feed data quick enough