I am assuming you are using a distributed launch. The warning message is self-explanatory. It seems like in each particular process other GPUs are still visible to the process. Ideally, on local_rank X, you want to only GPU X to be visible. There are some workarounds:

Set CUDA_VISIBLE_DEVICES = int(os.environ[“LOCAL_RANK”]) in your main worker function
use torch.distributed.barrier(device_ids=int(os.environ["LOCAL_RANK"])


In your case, RANK 6 is using GPU 0 for barrier but it should use GPU 6 as barrier.
You have to also be careful about how you set the device you want to use in your script. the set device should also be equal to the LOCAL_RANK.