I’ve just realized that we shouldn’t wrap the evaluation phase inside if torch.distributed.get_rank() == 0:, because data_loader_test also splits the data across all the processes.
And for the print() part, this code 10 explains why the message is displayed only once.
So now I understand what @fmassa did. Thanks.