Context-Aware Image Descriptions for Web Accessibility
<|reference_start|>Context-Aware Image Descriptions for Web Accessibility: Blind and low vision (BLV) internet users access images on the web via text descriptions. New vision-to-language models such as GPT-V, Gemini, and LLaVa can now provide detailed image descriptions on-demand. While prior research and guidelines state that BLV audiences' information preferences depend on the context of the image, existing tools for accessing vision-to-language models provide only context-free image descriptions by generating descriptions for the image alone without considering the surrounding webpage context. To explore how to integrate image context into image descriptions, we designed a Chrome Extension that automatically extracts webpage context to inform GPT-4V-generated image descriptions. We gained feedback from 12 BLV participants in a user study comparing typical context-free image descriptions to context-aware image descriptions. We then further evaluated our context-informed image descriptions with a technical evaluation. Our user evaluation demonstrated that BLV participants frequently prefer context-aware descriptions to context-free descriptions. BLV participants also rated context-aware descriptions significantly higher in quality, imaginability, relevance, and plausibility. All participants shared that they wanted to use context-aware descriptions in the future and highlighted the potential for use in online shopping, social media, news, and personal interest blogs.<|reference_end|>
@article{mohanbabu2024context-aware,
    title={Context-Aware Image Descriptions for Web Accessibility},
    author={Ananya Gubbi Mohanbabu and Amy Pavel},
    journal={arXiv preprint arXiv:2409.03054},
    year={2024},
    doi={10.1145/3663548.3675658},
    archivePrefix={arXiv},
    eprint={2409.03054},
    primaryClass={cs.HC}
}