Indeed, batch size 1 with DataParallel goes to first specified device (or defaults to cuda:0).
If you want to do inference with batch size 1 there is no need to use nn.DataParallel. This would be useful only if you have a much larger batch that you want to automatically split and automatically run on multiple GPUs. If you want to manually balance batches of size 1 you’re going to have to copy the model yourself and round robin over it. You’re right that the weights are not automatically updated if the source model is updated, because they are different tensors at that point. You’ll have to re-initialize the per-device models every time after running an optimizer on a single source model. In fact, this is exactly how nn.DataParallel works under the covers. On every call to forward it replicates a single module over N devices, scatters the input, runs forward on every one of them, and gathers the outputs. This repeats for every iteration.