Hi @pritamdamania87, thank you for your answer. I didn’t notice the gather_object because it is a new feature included in the current 1.8.0 torch nightly. Unfortunately, the gather_object doesn’t work with NCCL backend, so I have used the all_gather_object to broadcast the results to all the processes (and then use it only in the main one). I hope gather_object will be available for NCCL backend in the next release.
Thank you!