oh! Thats interesting. I’m using the SyncBatchNorm layer currently in single node 8GPU training.
Did you try to implement your own BN synchronization code? I see that you had an idea here



 weiwei:

so do we need to allreduce the model in the training


Did you implement synchronization? Also, were you successful in getting your accuracy up?



 weiwei:

(2)Furthermore, in the backward process, different bn parameters might cause the gradient calculated different in every layer, i am not sure whether the gradient is calculated in every each rank or they only broadcast the gradient calculated in rank 0.


Also, did you test the above? If you did, what were the results?