Hi,
I’m attempting to train my model over multiple nodes of a cluster, on 3GPUs. I am running the training script from Node 1, where GPUs 0, 1 are present while Node 2 has GPU 2. Following the distributed training example of FasterRCNN with this command, CUDA_VISIBLE_DEVICES=0,1,2 python -m torch.distributed.launch --nproc_per_node=3 --use_env train.py , I get the following error,
HCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=37 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "train.py", line 266, in <module>
    train_fpn()
  File "train.py", line 115, in train_fpn
    init_distributed_mode(args)
  File "/udd/rsundara/Code/head_detection/vision/utils.py", line 321, in init_distributed_mode
    torch.cuda.set_device(args.gpu)
  File "/udd/rsundara/.local/lib/python3.6/site-packages/torch/cuda/__init__.py", line 281, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:37

Can someone please help me figure out how to train models across multiple nodes?
Regards,