Human-inspired Explanations for Vision Transformers and Convolutional Neural Networks
<|reference_start|>Human-inspired Explanations for Vision Transformers and Convolutional Neural Networks: We introduce Foveation-based Explanations (FovEx), a novel human-inspired visual explainability (XAI) method for Deep Neural Networks. Our method achieves state-of-the-art performance on both transformer (on 4 out of 5 metrics) and convolutional models (on 3 out of 5 metrics), demonstrating its versatility. Furthermore, we show the alignment between the explanation map produced by FovEx and human gaze patterns (+14\% in NSS compared to RISE, +203\% in NSS compared to gradCAM), enhancing our confidence in FovEx's ability to close the interpretation gap between humans and machines.<|reference_end|>
@article{panda2024human-inspired,
    title={Human-inspired Explanations for Vision Transformers and Convolutional
  Neural Networks},
    author={Mahadev Prasad Panda, Matteo Tiezzi, Martina Vilas, Gemma Roig, Bjoern
  M. Eskofier, Dario Zanca},
    journal={arXiv preprint arXiv:2408.02123},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.02123},
    primaryClass={cs.CV cs.LG}
}