What kind of loss are you using? NLLLoss?
What are your output and target shapes?
Could you have a look at the error image ((prediction - target).abs())?
Do you see some errors at specific regions, e.g. corners?