Christian_Ses:

Usually the prediction is biased towards the biggest class, which is the background.


I imagine you are using Cross-Entropy loss somewhere. You could try to balance the class importance on the loss by setting different weights. Maybe this thread 10 could help a bit.