All good @Kiuk_Chung
We all are very busy in fact
I have not been able to solve it
Running on a single node I realized that it was not necessary to use mpiexec in front of torch.distributed.launch
The correct command to use in the script is the following

singularity exec --nv -B $imagenet_path $centos_path python3.8 -m torch.distributed.launch --nproc_per_node=2 $re_path --b 256 -f 2 -v $best_model_path $imagenet_path

Here I specify 2 processes per node since this node have two GPUs (--nproc_per_node=2)
I launch the scrip using the following line in the command line

qsub -n 1 -t 720 -A my-project-name ./run_Active_SimCLR.sh

Here I am asking for a single node (-n 1) for 12 hours (-t 720)
Remember that I have this running inside a container
As the documentation 4 points out, when I want to run the same code in two nodes I use the following
singularity exec --nv -B $mscoco_path $centos_path python3.8 -m torch.distributed.launch --nproc_per_node=2 --nnodes=2 --node_rank=0 --master_addr="192.168.1.1" --master_p    ort=1234 $cl_path --b 128 -t -v $mscoco_path &
singularity exec --nv -B $mscoco_path $centos_path python3.8 -m torch.distributed.launch --nproc_per_node=2 --nnodes=2 --node_rank=1 --master_addr="192.168.1.1" --master_p    ort=1234 $cl_path --b 128 -t -v $mscoco_path

The new options here that the documentation 4 specifies are the number of nodes (--nnodes=2) and the node id (--node_rank=0)
And I launch it by
qsub -n 2 -t 720 -A my-project-name ./run_Active_SimCLR.sh
Where I ask for 2 nodes during 12 hours
Using print statements I realized that the code gets stuck at this line
torch.distributed.init_process_group(backend='gloo', init_method='env://')
When I check the nodes using qstat they are still listed as running but they are like idle