Do you get constant predictions?
I’m asking because one of the things to keep in mind is that if you have very few “interesting” pixels in a large number of uninteresting ones, you essentially have a heavily imbalanced problem, and that can hinder training. To solve this, one could balance the pixels, e.g. when training the U-Net in our book, we take care that we have enough slices with nodules fed into the U-Net. A brief discussion is in section 13.5.5 (Designing our training and validation data).
Best regards
Thomas