Yes, i use every rank to validate a subset of my validation set, it is a default setting in distributed distributeddataparallel when we define the dataset loader.  Anyhow it is not an important point.
As the default setting of broadcast_buffers is True. so batch normalization is only calculated in rank 0 and every other rank shared the same batch normalization statistic in rank 0.
but in fact accuracy is different. (1)when i only save the  model in rank 0 and all the other ranks load rank0’s model to do inference accuracy is 75%. (2) each rank save its own model and load its own model to do inference accuracy can be 75.8%.
since model.eval() will freeze the bn parameter.
i think the most difference might be : (1) each rank’s model is not same. they share different model, such as bn parameters at least.
(2)Furthermore, in the backward process, different bn parameters might cause the gradient calculated different in every layer, i am not sure whether the gradient is calculated in every each rank or they only broadcast the gradient calculated in rank 0.
so, what is the truth ?  Thank you very much.