@miladiouss whats your batch_size?? If its like 32 and you are distributing the data to 2 GPUS, the effective batch_size would be 16 . Here, since the batch_size is small, the parameters of BatchNorm layers are highly biased towards the mini-batches. This might be the reason.
Try other norms like SwitchNorm in case of small batch_size