Solved by Diego in post #2 
              
              
                You are using nn.DataParallel so the batch is split evenly across all GPUs so the initial 64 batch size is split between both gpus into 32 and 32. I think that this might be the problem in your case.