Yes, nn.DataParallel would split the input batch in dim0 and transfer each chunk to the corresponding device.



 Hyeonuk_Woo:

If I use 4 devices, does the number A have to be a multiple of 4 for efficient calculation?


Also yes, but to get the most efficient approach you should use DisttributedDataParallel with a single process per GPU.