Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge
<|reference_start|>Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge: Large Language Models (LLMs) have demonstrated remarkable success in tasks like the Winograd Schema Challenge (WSC), showcasing advanced textual common-sense reasoning. However, applying this reasoning to multimodal domains, where understanding text and images together is essential, remains a substantial challenge. To address this, we introduce WinoVis, a novel dataset specifically designed to probe text-to-image models on pronoun disambiguation within multimodal contexts. Utilizing GPT-4 for prompt generation and Diffusion Attentive Attribution Maps (DAAM) for heatmap analysis, we propose a novel evaluation framework that isolates the models' ability in pronoun disambiguation from other visual processing challenges. Evaluation of successive model versions reveals that, despite incremental advancements, Stable Diffusion 2.0 achieves a precision of 56.7% on WinoVis, only marginally surpassing random guessing. Further error analysis identifies important areas for future research aimed at advancing text-to-image models in their ability to interpret and interact with the complex visual world.<|reference_end|>
@article{park2024picturing,
    title={Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge},
    author={Brendan Park, Madeline Janecek, Naser Ezzati-Jivan, Yifeng Li, and Ali
  Emami},
    journal={arXiv preprint arXiv:2405.16277},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.16277},
    primaryClass={cs.CL cs.AI cs.CV cs.LG}
}