@mrshenli IIUC singularity’s qsub command is invoked from the submitting process. @dariodematties I’m assuming you are using dist.init_process_group(backend="mpi")  inside $mscoco_path?
Since you are already launching with mpiexec there is no need to wrap your script with the distributed launcher. See: https://github.com/pytorch/pytorch/blob/49f0e5dfeb64a928c6a2368dd5f86573b07d20fb/torch/distributed/distributed_c10d.py#L446 13
The rank and world size information is provided by mpi runtime.