Do you mean that once it’s set, it cannot be changed?

I believe so.

For my case, I’m hoping to make GPU 0 visible on the 1st iteration, GPU 1 visible on the 2nd, etc till GPU 7 and iter 8. Is there a way to do this from Python?

Can this be done by explicitly passing torch.cuda.device(i) to tensors/modules or use torch.cuda.set_device(i)? Is there any reason that you would like to change the visible devices?