Thanks a lot for the quick reply. It was really helpful. I have understood how to use reduce functions to solve the issue and I can get correct metrics as I was getting for single GPU.
Could you please share if there is any way I can gather dictionaries created on multiple processes for multi-gpu on rank 0 process as for now I think all such reduce/gather functions work for only tensors?