How did you run the different processes?
Did you use CUDA_VISIBLE_DEVICES=ID in your environment or did you push the tensors to the appropriate GPU in the different codes?
If you’ve used the second approach, PyTorch still sees all GPUs and uses GPU0 by default to initialize CUDA as far as I know.