I managed to properly train semantic segmentation models with pytorch (but not yet on Pascal VOC).
I’d say that you should print during training the confusion matrix of the training/validation examples that you have seen, and then tune the learning rate / other parameters inspecting the training / validation metrics.
One more thing, I have the impression that you are not initializing your networks with a network pre-trained on ImageNet, is that the case? Note that in the FCN paper, they mention that they completely failed to train the network from scratch.