Hi @osalpekar, thanks for your response
I have figured out that the training runs fine when I remove the metric logger for MLMAccuracy 2
The place where it gets stuck is when the get() function 1 is called in  MLMAccuracy when writing to tensorboard and logger. Somehow it is not able to do all_reduce here for sum_metric 2  for this particular metric when more number of GPUs are there.
In case it makes sense to see the log files as you suggested:
I have posted the output with NCCL_DEBUG=WARN in 2 files here for 2 cases (freezes and works_fine): https://gist.github.com/amogh112/84a27280e69b983ea88497892e3855cb 4
And the comment at the end shows the output for NCCL_DEBUG=INFO for different cases with 1,2,6 GPUs