@Yanli_Zhao , I suggested to @Saichandra_Pandraju  to run this simple test:
echo 'import os, torch; print(os.environ["LOCAL_RANK"]); torch.distributed.init_process_group("nccl")' >  test.py
python -m torch.distributed.launch --nproc_per_node=1 test.py

and it hangs in his kubeflow environment, whereas it should work just fine.
This is the simplest reproducible script I can think of that doesn’t involve any other parts.
For context I have been trying to help @Saichandra_Pandraju in this Issue: Model Parallelism for Bert Models · Issue #10151 · huggingface/transformers · GitHub 8 but no matter what I suggested init just hangs there.
DeepSpeed on single gpu with ZeRO-Offload still needs distributed env, due to its implementation, so that’s why we are trying to figure out this problem. And I don’t think gloo which does work for @Saichandra_Pandraju  will work with deepspeed as it’s lacking features from nccl.

But reading his last follow up, once he matched cuda versions of pytorch and system-wide one the basic launcher now works. Which is odd that he needed to match the two, as pytorch’s distributed shouldn’t be impacted by the system-wide cuda install.
But then it gets stuck at a later stage will investigate the rest now.