Learning from Models and Data for Visual Grounding
<|reference_start|>Learning from Models and Data for Visual Grounding: We introduce SynGround, a novel framework that combines data-driven learning and knowledge transfer from various large-scale pretrained models to enhance the visual grounding capabilities of a pretrained vision-and-language model. The knowledge transfer from the models initiates the generation of image descriptions through an image description generator. These descriptions serve dual purposes: they act as prompts for synthesizing images through a text-to-image generator, and as queries for synthesizing text, from which phrases are extracted using a large language model. Finally, we leverage an open-vocabulary object detector to generate synthetic bounding boxes for the synthetic images and texts. We finetune a pretrained vision-and-language model on this dataset by optimizing a mask-attention consistency objective that aligns region annotations with gradient-based model explanations. The resulting model improves the grounding capabilities of an off-the-shelf vision-and-language model. Particularly, SynGround improves the pointing game accuracy of ALBEF on the Flickr30k dataset from 79.38% to 87.26%, and on RefCOCO+ Test A from 69.35% to 79.06% and on RefCOCO+ Test B from 53.77% to 63.67%.<|reference_end|>
@article{he2024learning,
    title={Learning from Models and Data for Visual Grounding},
    author={Ruozhen He, Paola Cascante-Bonilla, Ziyan Yang, Alexander C. Berg,
  Vicente Ordonez},
    journal={arXiv preprint arXiv:2403.13804},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.13804},
    primaryClass={cs.CV cs.CL cs.LG}
}