I need to calculate accuracy and I have 4 samples in total and 2 GPUs.
When I run testing I will have predictions and ground truths for 2 2 samples in each process.
Now if I want to calculate accuracy do I need to call dist.reduce or it is not needed and I can directly calculate accuracy in rank 0 process.

Do you need to calculate local metrics or global metrics? To calculate global metrics, you would need to communicate between the processes Distributed communication package - torch.distributed — PyTorch 1.9.0 documentation 12.