Could you post the (simplified) code, so that we could exclude some coding errors?
As a first step in debugging I could suggest to use only one single training sample and try to overfit your model badly on it. The loss should decrease to zero and the segmentation should look clear.
If that’s not possible, your model or training procedure might have some bugs.