Thanks @Vikas_Sangwan and everyone else. So, my batch size is 64 and I had 2 classes and 2 GPUs. And it seems the issue is from batch normalization. For now, I am shuffling my minibatches so each GPU gets the same number of each class. I will try other normalization methods and write an update at some time. Thanks for the help everyone.