I does not look like you are getting gradient vanishing. In fact it is difficult with unet due to skip connections. It also looks like that even decreasing lr you are arriving to same local minima, thus, a bigger LR at early stage does not help not to fall in there. Could be a matter of trying different initialization/optimizers in order to see if the lead to another place?
Anyway, do you have a healthy dataset with balanced amount of classes?