CV-Probes: Studying the interplay of lexical and world knowledge in visually grounded verb understanding
<|reference_start|>CV-Probes: Studying the interplay of lexical and world knowledge in visually grounded verb understanding: This study investigates the ability of various vision-language (VL) models to ground context-dependent and non-context-dependent verb phrases. To do that, we introduce the CV-Probes dataset, designed explicitly for studying context understanding, containing image-caption pairs with context-dependent verbs (e.g., "beg") and non-context-dependent verbs (e.g., "sit"). We employ the MM-SHAP evaluation to assess the contribution of verb tokens towards model predictions. Our results indicate that VL models struggle to ground context-dependent verb phrases effectively. These findings highlight the challenges in training VL models to integrate context accurately, suggesting a need for improved methodologies in VL model training and evaluation.<|reference_end|>
@article{beňová2024cv-probes:,
    title={CV-Probes: Studying the interplay of lexical and world knowledge in
  visually grounded verb understanding},
    author={Ivana Bev{n}ov'a, Michal Gregor, Albert Gatt},
    journal={arXiv preprint arXiv:2409.01389},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.01389},
    primaryClass={cs.CL}
}