I just found out that LayerNorm with learnable parameter destroy the synchronization between the weights.
SyncBatchNorm is a solution for BatchNorm, but LayerNorm doesn’t have SyncLayerNorm. I just give up using DistributedDataParallel because it may not improve the performance if I add more synchroize point for LayerNorm. In my case, almost all submodule have LayerNorm.
By the way, I really appreciate your help and thank you.