It’s hard to answer which architecture will definitely work better.
Currently your model seems to fail at the segmentation task.
However, the model might still be a good fit, but the hyper-parameters, e.g. learning rate, are messing up the training.
Have you checked your per-class accuracies? Most likely some classes will be completely ignored.