The only difference I can imagine is that the BN parameters don’t get replicated from rank 0 to the other ranks after the final call to backward, since they are frozen for evaluation after that. I highly doubt that can be the cause of a 0.8% accuracy difference though. What is your approach for aggregating the accuracy numbers?
Regarding (2): the gradient is computed in every process and then averaged across them. This means that if you start with identically initialized model weights and use the same optimizer in all processes, the optimizer will take the same step everywhere.