You could try a weighted loss using the inverse of the class frequencies, which might help if your model is overfitting.
However, I’m wondering why you’re getting a high loss for patches containing just background.
If your model overfits to the background class, these patches should be nearly perfectly classified.