To get mean metrics all across the ranks, I use all-reduce function something like this:
import torch.distributed as dist

def global_meters_all_avg(args, *meters):
    """meters: scalar values of loss/accuracy calculated in each rank"""
    tensors = [torch.tensor(meter, device=args.gpu, dtype=torch.float32) for meter in meters]
    for tensor in tensors:
        # each item of `tensors` is all-reduced starting from index 0 (in-place)
        dist.all_reduce(tensor)

    return [(tensor / args.world_size).item() for tensor in tensors]

See Distributed communication package - torch.distributed — PyTorch 1.8.1 documentation 3 for more details about the collective communications.