@ayalaa2 thanks for the prompt reply!
Does it mean I need a distributed sampler on the validation set as well?
So, if I want to measure accuracy on my validation, each rank should count the number of “correct” predictions and the number of samples it saw and the “reduce” it?
Would that look something like this:
def validate(val_loader, model):
  # one counter for "correct" the other for "total"
  counter = torch.zeros((2,), device=torch.device(f'cuda:{args.rank}'))
  model.eval()
  with torch.no_grad():
    for x, y in val_loader:  # assuming val_loader has a distributed sampler
      pred = model(x)
      num_correct = my_accuracy_function(pred, y)  # count number of correct predictions
      counter[0] += num_correct
      counter[1] += x.shape[0]  # total number of samples 
  # done validating for this view
  torch.distributed.reduce(counter, 0)  # reduce all to rank 0 process
  if args.rank == 0:
   # only this rank reports accuracy
   print(f'total accuracy = {counter[0]}/{counter[1]} = {counter[0]/counter[1]}')

Does that make any sense?