If you see these in training, too, you could make a point of oversampling them in the next epoch (or an intermediate mini-epoch). More elaborate schemes of training instances where the model gives undesired results go under the name of “hard example mining”.
Another option could be to add a post-hoc classification step where you train a classifier to have a second opinion about whether the detected image shows anything.
Best regards
Thomas