Say my code is main.py. I create a tmux session and 8 panes in it. In i-th panel, I run 'python main.py --gpu_idx i" where i starts from 0 to 7 and --gpu_idx indicates the device id of the gpu I want to use. At the very beginning of the code there is torch.cuda.set_device(args.gpu_idx) to set the gpu to use. Then, I apply .cuda() to the network and loss function, and in the training loop, I wrap the data and labels tensors as Variables, and do data = data.cuda() and labels = labels.cuda(), in a very standard way.  I never used CUDA_VISIBLE_DEVICES=ID. So, I think I used the second approach you are mentioning. But, what is really strange is that in the other machine, the very same code works just as expected, meaning that running main.py --gpu_idx i only creates a process in the corresponding gpu, and not in GPU0.