As far as I know, The states that I need to move include : weights, bias,running_mean,running_var, is there any else that I need to transfer over ?
Mentation about replacing SyncBatchNorm layers, could you provide simple example how to do this, maybe something like :
for  name , module in model.named_modules():
    if isinstance(module, nn.SyncBatchNorm) : 
        #do something