If it’s the case, are you sure you are choosing the correct devices? you can see if you can manually allocate tensors
a=torch.rand(100).cuda(idx) manually changing idx
and check if the idx match the nvidia-smi or in your case the pci order