Thanks for the information!
Let’s try to narrow down the source of the problem.
Are other processes working on the GPUs?
Are you able to create a tensor on all devices?
If so, is nn.DataParallel running successfully?