The result is as shown:
without os.environ[DEVICE_ORDER]=PCI_BUS_ID,torch.cuda performs same device order as nvidia-smi order.
with this env variable, the cuda and nvidia-smi order is changed
cuda(0)----gpu2
cuda(1)----gpu3
cuda(2)----gpu0
cuda(3)----gpu1

and the same error reports.