I made some changes to the model’s forward pass in VL-BERT repository 1.
I was able to successfully run my script for training over multiple (7) GPUs. However after some time, suddenly, my code freezes when I increase the number of GPUs to more than 2 GPUs. I did not make any changes to the way model is passed for distributed training.
This is how I increase the number of GPUs used:
CUDA_VISIBLE_DEVICES=1,2,3,4 ./scripts/dist_run_single.sh 4 pretrain/train_end2end.py ./cfgs/contrastive_pretrain/base_prec_random_movienet_images_4x16G_fp32.yaml ./checkpoints_debugcv04
Gets stuck if GPUs are more than 2
instead of
CUDA_VISIBLE_DEVICES=1,2 ./scripts/dist_run_single.sh 2 pretrain/train_end2end.py ./cfgs/contrastive_pretrain/base_prec_random_movienet_images_4x16G_fp32.yaml ./checkpoints_debugcv04 
Starts training successfully
The model is loaded on each of the local ranks successfully.
The script also enters the train function on each rank : https://github.com/jackroos/VL-BERT/blob/4373674cbf2bcd6c09a2c26abfdb6705b870e3be/common/trainer.py#L56 2
However, the forward pass doesn’t proceed.
I am using the latest version of PyTorch 1.7.0
What might be going wrong here? I assume some synchronization problems might be occurring with more than 2 GPUs
Thanks