Based on the target map you’ve posted it seems you are dealing with a heavily imbalanced dataset, i.e. the majority of all pixels belongs to the background class while only a small fraction represents the other class.
The model might thus easily overfit to the background class, as it’ll decrease the loss easily and also would yield a high accuracy.
You could try to use e.g. a weighted loss in order to force the model to learn class1.