I also ran into similar issue that  4 gpu with  4 x batch_size   almost cost 4 times processing time of  1 gpu with 1x batch_size iteration.
For this statement:



 ptrblck:

use DistributedDataParallel with a single process per device


Could you give an examples?
I am not very sure what "single process per device " mean…