How do the two virtual machines connect to each other and run GPUs from another machine parallelly? Out of two virtual machine, on which machine should I run the below distribution command?
python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_YOU_HAVE
–nnodes=2 --node_rank=0 --master_addr=“192.168.1.1”
–master_port=1234 YOUR_TRAINING_SCRIPT.py (–arg1 --arg2 --arg3
and all other arguments of your training script)
python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_YOU_HAVE
–nnodes=2 --node_rank=1 --master_addr=“192.168.1.1”
–master_port=1234 YOUR_TRAINING_SCRIPT.py (–arg1 --arg2 --arg3
and all other arguments of your training script)