I don’t know which criterion you are using, but adding the dice loss might help for segmentation use cases.
Also, you could try to add skip connections into your conv blocks, which seemed to work out pretty well in the past.