I only encounter it in a distributed setup. Either running the code on two GPUs or four GPUs meets the same problem. For a single GPU, there is no problem. A quick code to reproduce shows below:
import torch
torch.cuda.set_device(0)
torch.distributed.init_process_group(backend=“nccl”, init_method=“env://”)
torch.distributed.barrier()
And the running command is: CUDA_VISIBLE_DEVICES=3,4 python -m torch.distributed.launch --nproc_per_node=2 quick_code.py
where quick_code.py contains the four-line code.
The problem appears in V100 GPUs. However, the same code is OK when I run it on GeForce GTX-1080 GPUs or RTX-2080 GPUs.