Hi,
I’ve successfully implemented semantic segmentation models in pytorch, so I think that should not be a bug in PyTorch.
So, there a few things I’d check in your code:

it seems that you use inputs in the range 0-255, without any normalization. is it intentional? I’d at least put them in the 0-1 range, and eventually subtract 0.5 or the dataset std.
batch normalization with batch size 2 doesn’t look right to me, might introduce a very large bias because the batch size is so small. I’d say to use batch norm sucessfully you’d need batch sizes of 128 or 256
it seems that you don’t use pre-trained networks for initializing your model? This could lead to reduced performances