Solved by ptrblck in post #2 
              
              
                If you want to check how the devices are connected inside your machine, you could run nvidia-smi topo -m in your terminal. 
I might have misunderstood the question, but nn.DataParallel replicated the model on each device as explained here.