WIth two gpus for the same batch size, your batch is divided in two, so the batch size seen by each gpu is two times inferior by the one seen by your single gpu in your first experiment. Since batch norms tensors are specific to a gpu, this means that the batch size seen by batch norm layers is divided by two. This could explain the difference.