Wouldn’t an all background input predicted as such yield TN = 1?
If the model predicts another class and the dice loss shoots up (you might need to limit it in case it goes to Inf), the model will get a high signal to predict background only.
Do you mean that the very high dice loss biases the model to only predict all background images?
If so, you could try to clip the loss or just remove these images.