I assume that you are talking about PASCAL VOC 2012 segmentation dataset.
In my view, it depends on how you are calculating the loss. When you are using CrossEntropyLoss or NLLLoss2d (per pixel), you should not normalize the ground truth.
What kind of loss function are you using?