@bodokaiser the lack of any convergence even on train might be some bug (maybe even not in the network but in how inputs/outputs are prepared) or bad learning rate, and it can very much depend on the dataset - sorry, don’t have any more insights about this.
One thing that is different in my implementation is that I use upsampling instead of transposed convolutions, it worked significantly better in my case. Batch normalization speeds up convergence but is by no means essential, it worked fine without it too. Simpler models also gave okayish results, but UNet was consistently better - in this task the metric was intersection over union, and simple models were giving results in the 0.2-0.3 range (average over 10 classes), while UNet gave 0.4+ without too much tuning.