The difference between BatchNorm and SyncBatchNorm is that SyncBatchNorm uses torch.distributed.all_reduce in the backward pass.
Two questions:

What args and env vars did you pass to init_process_group?
If you program, is there any other code that launches communication ops?