  Deep neural networks have exhibited remarkable performance in various
domains. However, the reliance of these models on spurious features has raised
concerns about their reliability. A promising solution to this problem is
last-layer retraining, which involves retraining the linear classifier head on
a small subset of data without spurious cues. Nevertheless, selecting this
subset requires human supervision, which reduces its scalability. Moreover,
spurious cues may still exist in the selected subset. As a solution to this
problem, we propose a novel ranking framework that leverages an open vocabulary
object detection technique to identify images without spurious cues. More
specifically, we use the object detector as a measure to score the presence of
the target object in the images. Next, the images are sorted based on this
score, and the last-layer of the model is retrained on a subset of the data
with the highest scores. Our experiments on the ImageNet-1k dataset demonstrate
the effectiveness of this ranking framework in sorting images based on
spuriousness and using them for last-layer retraining.

Computer Vision and Pattern Recognition;Machine Learning