Solved by gcramer23 in post #2 
              
              
                I need to calculate accuracy and I have 4 samples in total and 2 GPUs. 
When I run testing I will have predictions and ground truths for 2 2 samples in each process. 
Now if I want to calculate accuracy do I need to call dist.reduce or it is not needed and I can directly calculate accuracy in rank …