DEAL: Disentangle and Localize Concept-level Explanations for VLMs
<|reference_start|>DEAL: Disentangle and Localize Concept-level Explanations for VLMs: Large pre-trained Vision-Language Models (VLMs) have become ubiquitous foundational components of other models and downstream tasks. Although powerful, our empirical results reveal that such models might not be able to identify fine-grained concepts. Specifically, the explanations of VLMs with respect to fine-grained concepts are entangled and mislocalized. To address this issue, we propose to DisEntAngle and Localize (DEAL) the concept-level explanations for VLMs without human annotations. The key idea is encouraging the concept-level explanations to be distinct while maintaining consistency with category-level explanations. We conduct extensive experiments and ablation studies on a wide range of benchmark datasets and vision-language models. Our empirical results demonstrate that the proposed method significantly improves the concept-level explanations of the model in terms of disentanglability and localizability. Surprisingly, the improved explainability alleviates the model's reliance on spurious correlations, which further benefits the prediction accuracy.<|reference_end|>
@article{li2024deal:,
    title={DEAL: Disentangle and Localize Concept-level Explanations for VLMs},
    author={Tang Li, Mengmeng Ma, and Xi Peng},
    journal={arXiv preprint arXiv:2407.14412},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.14412},
    primaryClass={cs.CV cs.AI cs.LG}
}