So you are using BCE with logits rather than crossentropy right? Have you tried to penalize the weight of background? It sounds like dice loss works as its kind of IoU but with crossentroypy model starts predicting all as background and it would be accurate loss-wise.