Hyeonuk_Woo:

If I use 4 devices and dim0 of my input data is 8, in this case, each GPU handle two process simultaneously or sequentially?
+) the word ‘single process’ means that batch size of the data assigned to single GPU is 1?


No, each device will get a batch containing 2 samples.



 Hyeonuk_Woo:

Why single process per GPU is recommended?


DDP should be faster as it reduces the communication overhead in nn.DataParallel.
The details of the latter (including the scatter/gather calls) are described in this blog post 1.