Could you describe your dataset more? I’m going to assume that by “background or not labeled objects”, you mean images that are labeled as “no object”.
Overfitting can be caused by a few issues, I would perhaps suggest balancing your dataset such that the number of images that are background images are balanced with images containing objects. However, as a result you may not have enough data to train on. You could then use a model that was pretrained on a similar task and fine-tune it with your training set.
That would be my immediate thought, but there’s other ways of tackling this issue.