I am updating my training script to use Distributed Data Parallel to do Multi-GPU training.
I am done with most of the steps as mentioned in PyTorch Guidelines.
But I am confused about how to handle metrics calculation and visualization:
For example,
I need to calculate accuracy and I have 4 samples in total and 2 GPUs.
When I run testing I will have predictions and ground truths for 2 2 samples in each process.
Now if I want to calculate accuracy do I need to call dist.reduce or it is not needed and I can directly calculate accuracy in rank 0 process.