The problem isn’t that pytorch uses all GPUs by default (it’s not true, everything is put on device 0 by default), but that you scattered some tensors on different GPUs and then tried to do operations on them. If you want to do that you need to insert the .cuda(destination) calls to move tensors to the appropriate device.