It may be beneficial to perform binary classification on the image to better determine if the object you are interested in segmenting is in it or not. If you are using a U-Net architecture, you can do both classification and segmentation at once by passing the output of the middle layer (which contains the context of the input) into a single output dense layer with sigmoid activation. Any positive pixels in the mask generated by the second half of the U-Net can be disregarded if the binary classification deems that the object is not in the image. I used this technique in a pneumothorax segmentation competition hosted on Kaggle, and it improved my score.