According to the tutorial 18, DataParallel splits “data” automatically across available GPUs. I’m pretty sure it only works on batches, so you need batches of more than 1 sample, otherwise it might (a) make no sense to split data, (b) be very inefficient due to synchronization…
Do you even have any usage on other GPUs than the first one? If you have batch sizes of 1, nothing would be split across GPUs and only CUDA:0 would be used.