Hi @mrshenli, thanks for your reply.
I tried with  “gloo”, it does terminate the process and throws Exceptions. But with “nccl”, I add the following line in the bash script to run the .py file,
export NCCL_BLOCKING_WAIT=1

However it doesn’t work.
Also tried with adding the following line in the .py file itself, but doesn’t work either.
os.environ["NCCL_BLOCKING_WAIT"] = "1"