I’m asking because we had this problem recently when a model without SyncBatchNorm was getting deadlocked during boradcasting of buffers.
The reason of the deadlock was different: Checkpointing may cause the NCCL error · Issue #1166 · speechbrain/speechbrain · GitHub 1, but it was still strange that it still went into the code path of broadcasting the buffers. In debugging the buffers were the stats of regular BatchNorm1d.